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ABSTRACT 

A 2-day seminar, attended by 35 physicians and 
others concerned with graduate physician education, was held to 
review the state-of-the-art of evaluation, current techniques, and 
effectiveness of programs. Presentations during the opening session 
were: (1) "The State of the Art of Evaluation" by Ray Elling, (2) 
"Defining Objectives in Evaluation" by George E. Miller, (3) 
"Contemporary Activity in Program Evaluation" by John P. Hubbard, (4) 
"Judgement Evaluation" by Frank L. Husted, and (5) "Whose 
Evaluation?" by Edwin F. Rosinski. The second session was a 
round-table discussion of evaluation programs used by divisions of 
the Association for Medical Education and identification of 
commonalities in these programs. Some conclusions were: (1) There is 
need for more relation to society and also for more effective 
evaluation techniques, (2) Evaluation should be a continuum as is 
education, and (3) There is a need for more resources in evaluation. 
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FORWARD 



Periodic evaluation ‘of activities offers an opportunity for 
recognition of the productive and nonproductive and for retention, 
improvement or discard of those activities. The traditional func- 
tion of evaluation in education, particularly medical education, has 
been for individual accreditation. But evaluation of participating 
individuals in medical education can also serve to measure the 
process of education. ; >:i 

This conference on Evaluation of Graduate Physician Man- 
power Education was designed to provide a review of the state of 
the art of evaluation, current techniques, and the effectiveness of 
programs. The key to progress in graduate medical education is 



mits constructive change. 
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CHAIRMAN HALL: We want to maintain a considerable amount 
of informality here. I think that everybody knows who they are and I 
suspect most of us know who everybody else is. I think in maintaining 
the informality and the exchange we would like to have, I would like to 
have everyone say who they are and who they identify with, and we'll 
go around the table this way. 

. . . Participants introduce themselves . . . 

CHAIRMAN HALL: It is almost redundant to say that We have 
probably the busiest people in graduate.; medical education gathered 
here this evening, and we are deeply honored that you Will spend your 
time with us and we are sure that you will give a great deal to us, 
and I hope that this will be meaningful for each of us. 

v I think We have gathered together here the essential enzjrme s 
to work with the substrates that each of us relates to. 1 I really look 
forward to a great deal of productivity in the nekt four or five years; 
from this C onference on Evaluation of Graduate Medical Education.45 

In the present state of the art in graduate medical education, ; 
the merit of a program seems more related to the Wocalhess of the • i 
directors — their ability to describe their program in glowing terms — 
than to fact. < We all (too often have measured the end prodUct of the 
educational process. We should have been : measuring the education K-- 
planning, or improvement in the individuals' that we are trying to gain; 
impr ovement in. If we are going to make' prbgres sllh graduate e ducar 
tion that the public will demand of us, we must have the dynamic process, 
with individuality built into it, that lets us change from year to year. 
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Before we can do this, we must have effective evaluation 
processes. Many of us in this room are engaged in evaluation 
procedures now. In the exchange of ideas that we gain over the 
next day and a half, each of us will be able to go home and do a 
more effective job. Tomorrow, in our last hour, I would like to 
ask each of you to bring forth the commonalities that we find that 
attribute to the success of the programs that will be discussed 
over the next day and a half. 

I would like now to introduce Dr. Angelides, President of 
the Association for Hospital Medical Education. 



Comments 

DR. ANGELIDES: At the present time AHME represents 
about 500 hospitals which are involved in medical education -- 
undergraduate, graduate and continuing. Our emphasis in the past 
has been primarily in the graduate field. This translates to roughly 
half the graduate programs in the United States. This is ebbing 
somewhat and concern with continuing education is coming to the 
fore. ••• \v • -• ^ --sV.V*.- ' »' •' /••• 

The questions often asked are : "How do I know what I am 
doing is correct? How do I know if I am doing a better job than other 
hospitals?" The purpose of the Conference is to bring together a 
representative group with an interest in some phase of medical educa- 
tion and an interest in evaluation of the programs for wtiic h e ac h is . , . . : 
responsible . { We think this is an appropriate time to find out what you 
are doing so we can learn, and hopefully others will learn at the same 
time.' . ’ ••/£:/ .'sV'-'-. j iV.* ;• i Vr f?:<yjcVx v.* :••** 

CHAIRMAN. HALL : < I want to s et down some ground rule s : one 
is informality; two, the identification of the individual. We are . record- 
ing, this, because we hope to gain some guidelines, some general informa- 
tion, that can be made available to people interested in evaluation and 
the state of the art in graduate medical education. I hope that this does 
not stifle the free flow of discussion, that people do not feel they have 
to speak in. a formal rt/.-V 

I would like to . star ; t now by asking Dr. Ray Elling, Professor 
of Sociology, Department of Clinical Medicine arid Health Care, : at the 
Uni ye r s ity of C onnecticut;. Health C ente r, to di scus s w the topic , > '.(The ; ;■! 

.State> bfc^be ; ^ : i ;£ ^ ~ • ' 

v:.':d:V‘5 
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: 'The State Of The Art Of Evaluation ." 

DR. RAY ELLING: I am a little overwhelmed by the size of 
the topic I have been assigned: "The State of the Art of Evaluation. " 
Needless to say, in fifteen minutes and with my limitations, I am not 
going to cover it all. I hope to select some of the highpoints of the 
climate and the context of evaluation. 

I want to ask the : question, "Are we ready for serious evalua- 
tion? " Then I want to consider some of the conflicts in the values that 
are involved in the evaluative process itself. Then I would like to talk 
just a little bit about the structure or organization of the evaluative 
efforts. f :• ; v : :..1 * • -x > -.U .; hiy •. «. • y-, 

^ In doing this I hope to save you from consideration of methodo- 

logical niceties, consideration of scientific methods, and the latest 
x. techniques for evaluation. I don't want to ignore it entirely though and 
I would like to point you toward Dr. Miller's very fine article on the 
orthopedic training program in the October, 1968, issue of JAMA. It 
indicates several new approaches toward evaluation: first, the samp- 
•y. ling of critical incidents, and second, th ; use of these materials in the 
.v development of a performance profile. T.xis performance profile employs 
several different tests and approaches, and focuses on the recall of 
information; the ability to observe, analyze, and interpret data; the 
ability to solve clinical problems; the ability to communicate effectively 
with patients and with colleagues; and the ability to exhibit skills that 
are essential to the particular specialty, in this case, orthopaedics. 

xis . The first point with respect to climate, are we., ready for serious 
: evaluation? Let me quote from E. Suchman's book: 

"The need for evaluation rarely occurs in an atmosphere of 
complacency. Dissatisfaction, and puzzlement lie behind most 
: .jZ-'44s (: -demands, for evaluation; " • ' >£ y 

isn't any place for evaluation effort in .a situation in which people 
l & are satisfied with what they are doing. r^There . would be too much 
/y ■.%! resistance to careful and systemaiic>evalua.tive effo^t.^So one needs 
either to enter into or create a situation in which there, is dissatisfac- 
v ■ tion with the way things are, and puzzlement as to how the' gaps can be 
•• x- closed between ideals and- realities|^yy|§^^ • y .• yi-:- 

.^yyyayyLet me test how ready we are by ; asking a few questions that 
v i U:::t:-.might.^suggest, s omemewi directions k yifcwi .'llib^aLbie^tq AelT p^lra 

expressions on your faces whether I am getting into touchy ter ri- 
;;4v.v ■ ;■ < ;xx; tory , ; or whethe r this is something ;ipld| : hafe^ :-y y]-?- 
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First of all, should universities be the ones to structure and 
provide the curricula in graduate education? It seems to me that 
there is a climate developing/' reflected by the Millis Report, reflected 
by the regional medical programs. In those programs one of the cen- 



tral problems we face is to connect the knowledge -developing and' 
knowledge -transmitting center with the periphery of the health system - 
namely, the place of practice where it occurs, and wherever it occurs. 
It seems to me that the history of graduate education in the community 
hospital is an admirable one but one in which there is some possible 
isolation from the center of knowledge development and transmission, 
which is the University Health Center. , •• v 

In any case, it strikes me that this is a question central and 
necessary to ask and evaluate: What about a program in which the 
connections are firm and the curriculum is established by the univer- 



sity, compared with a hospital, say;f in York, Pennsylvania,; in which 
setting the curriculum, and carrying out the program is not ; structured 
that wa 



vvU Another possible new direction to be considered is related to 
my own field of interest -- 1 the social sciences. I see the possibility 
of moving beyond the integration of basic science material in the treat- 
ment process to the integration of information from the social sciences 
This strikes me as relevant in two ways. • :4.e cuS 



■ First, I would think; that the introduction of perspectives . 

obtained through social-psychological studies, may be important fcr 
training in the judgment process itself . Not the content of clinical 
judgment and practice", but, rather, in the judgment process itself. 
Simply the awareness of the way in which men influence. one another 
in the process of reaching a judgment strikes me as an important 
content are a> it sy ■■■■?}. 3 : • 



.•tv 



There is a range of ambiguity that applies to questions like: 
What does this patient have as his problem? What is the appropriate 
course of continuing tests eind treatment ? ' There is a range of ambi- 
guity in whichisocial influence comes to bear. I don't see too much 
of the knowledge ■ available being introduced either at the graduate level 
;or at Any -othenlevel in ih^ today. 

In a second way it strikes me that; material from the . social 
sciences is relevant. And here what I am suggesting is with respect 
to the content of treatment itself. > The following kinds of questions 
seem! important to me: What about the patient as a s ocial being ? L 
How does one treat this person- adequately without iinder standing the 
process of s oc ial i z ati on ? / W hat ab out / the social/ cultural environment 



of the patient, the place to which, the person must return after treat- 
ment ? If a person has a boil on his butt and he is sent home to soak 
his posterior in a bathtub and he doesn't have a bathtub, it seems to 
me that you have a problem, and I am not sure that the educational 
processes take account of this aspect of care and treatment. 

What about the .economics of , care ? The structure and opera- 
tion of social power in the acceptance or rejection of new technologies 
and new forms of organized care ? What about the epidemiology of the 
problems that the intern and resident see ? What are the patterns of 
utilization of care ? What about health manpower problems, and, pre- 
ventive aspects of care.? To what extent does the content. of graduate 
x — j- x_ xi____ Matters ? 



From Dr. George Miller's article there comes forth a 
fascinating question which will really test the vested interests in the 
field. Should we shorten training ? Maybe one year is enough before 
one is a specialist. ; ; Looking at the graphs in that article, I see certain 
increments of knowledge between the first year and the second, and 
the second and the third, arid the third and fourth in some cases, but 
...' is the increment worth it ? In fact, in certain cases it seems to go the 
other way. The, level of knowledge is higher in the first year than in 
s ub sequent : ye ar s . Will, I 'm simply s aying, are we ready, for se rious 
evaluation ? If we are, we are ready to undertake the study of those 
kinds of ; questions and others; iV /. ~ ' 

Another point I Would like, to make with respect to the climate 
of evaluation has to do with the goals of care. I am afraid I infringe 
here a bit on subsequent speakers^, but I can't resist the temptation to 
suggest that thepe is a real convergence today between the goals of the 
e due a tor and the goal s. of. the person ponce rhea with, organi zing c are . 

• ; In closing, let me raise, finally, the question of conflicting 
values. The evaluative process represents an instrument of social 
power. It serves to re -combine, to re-define, to challenge, even to 
t hr e ate n, Zto .open the w ay to some but. not to others! ? What I am saying 
is that. we need; to consider, the, composition of the evaluating groups, 

.and I would like to leave with you, the questions : Should the evaluating 
i); group include some of the public., in it ? , Should we go beyond pur narrow , 
.vested, professional boimdaries, and consider introducing an element of 
public accountability in the evaluative .process ? And, finally, should we 
also have in the evaluation proce s s a kind of Mari from Mar s who may 
he cultural anthropologist visiting; a primitive, tribe, examining 



in. a sense; 

:pnre search, just .tp keep everyone honest in the process ? jV'; ,; 
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CHAIRMAN HALL: The floor is open for discussion. I will 
start out by raising this question: Evaluation, as you say, offers an 
instrument of change that can be helpful or can als o be threatening; 
but doesn't it also offer the possibility of gaining change by evolution 
rather than revolution? 

DR. ELLING: Yes. I think the process of evaluation, carried 
out carefully and systematically, with the grammar of science in mind, 
can be all important in introducing new approaches. It is like an 
engineering process in the basic sciences. It is a way of Structuring 
behavior so it reflects the knowledge one has in hand,- and, indeed. 



QUESTION: You raised the question but you didn't provide a 
suggestion about the answer on the last issue. Should we involve the 
public in our evaluation processes? ^ v ■ 

As a sociologist, as a social scientist, what is your view ? 

’ in knowing. 



DR. ELLING: I probably come to this question through some 
studies we have done of different professional groups in the public 
health field. Some twenty or so different groups were included in our 
study and, among other things, we asked them to rate each other on 
the extent to which they were professional. This included the physi- 
cian, the social worker, the sanitarian, the public health nurse, 
hospital administrator, the statistician, and so on. V' 

There were various ratings offered by each member of each 
group about other groups, and their own group. Some groups came 
out as highly professional; other groups came out as not so professional 



We thought we would look for the correlates of professionalism. 
Who is professional arid who is n j t ? f : By this mode of analyzing the •/ 
problem, we thought that things like service orientation, arid autonomy 
in parrying out one's work, arid things like this would be important. 
They didn't correlate at all. ’' Instead, the things that really mattered 
were income; ; years of education;’ which .could be a knowledge kind of 
factor but also might just be a union card, and sex. ;• Groups more male 
in composition, were seen ds rriorepr^ ■ 

In thinking about this whole process of be corning professional, 

■5 4' l’ • 1 f— / ■■ y 1 v *-1 ■ • -i '• . '& •*•■ *• j j '• ; _ j 1 1 ‘111 



itvStrikehyme t}i .t members.of - 

themselves and provide a place for themselves under the sun, and in 
doing this, they often forget at least parts of their overall re sponsibilities 
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to society. So I would tend to favor at least some experiments where 
public members were included in the evaluative process. 

DR. GEORGE MILLER: You would reject then Fie xner's early 
definition of the profession, of which the first criterion was that it 
determined for itself what was professional? 



DR. ELLING: One of my mentors, Professor Hughes, sociolo- 
gist at the University of Chicago, used to say that one definition of a 
profession is a group that reserves to define for itself its own mistakes. 
But it strikes me that with the rising demands for care and the defini- 
tion of it as a basic human right, and the rising curve of medical costs 
and the increasing influence of large third party payers, that this 
business of guarding the jewels in the center of the temple and not 
exposing them to anyone has more or less seen its day. 

DR. WINSTON MILLER: I guess the majority of us are in my 
era, about twenty years out of me die aL school. We didn't have the 
advantage of behavioral sciences when we were in medical school. We 
find now that we should let behavioral sciences control evaluation of 
what we do. We read the term "socio-economic medicine" lately. I 
know sociologists and I know economists but I didn't know either one of 
them got together to form a division of socio-economic medicine. We 
are talking about evaluation of educational programs in medicine, 
theoretically graduate and post-graduate...^;^ £ T' 

What does this all mean, and how can the disciplines of socio- 
econbmic medicine assist medicine in these problems ? . 

;'?/^'^'-..P^»^EL'LING:-: I think in a couple of ways, but I don't know that 
there is anything that unique other than a new point of view. I think 
that many social scientists would have a good bit to learn about the 
scientific method and its use in the clinical arena from physicians. 

But I suppose the one primary contribution would be in respe ct to pe r - 
Spective. And I simply mean by perspective the business of looking at 
people and their ways of behaving as a problem for study itself, just 
as the physician in the laboratory would look through a microscope at 
^ various clustemngsipl-bacteria or other forms of other phenomena. 

. . Attention to that kind of study perhaps does develop another perspective. 

n' ? DR. BUTT: Dr. Ellingi^I enjoyed very much your comments. 
Concerning your statemerit about guarding '.'the jewels of the physician, •' 
I think this may be not quite fair, because I don 't think !• really they have 




to judge 



T a physician, for his competence. You admitted in y pur experience that 
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it had no correlation, and I wonder if this is really possible, since we 
see sociologists and others sometimes pick the very worst physician. 
(Laughter). .. v . 



DR. ELLING: I think we have to start spelling out what we 
mean by the dimensions of good care, and it seems to me that there 
are some helpful leads in the work of Osier Peterson, where he is 
examining the logic of the care process itself, and is essentially asking 
three questions about this process: What is done that shouldn't have 
been done according to best current standards arid practice ? What is 
not done that should have been done ? And of the things that are done 
that should be done, how do they fit together ? How do they relate one 
to another, sequentially ?; "V 



I think quality care is very important, but I wonder if a sort 
of in-house examination of this process is entirely sufficient. For 
example, I would add some other dimensions to good care. I would 
ask the extent of which the care process leads to a degree of increased 
sophistication on the part of the recipient of care, with respect to 
health problems and what to do about them.- It strikes me that if the 
System were functioning adequately, it would attempt in part to turn the 
job over 'to the recipient popfaatio^^^ : ' 



" ’ I think one might also ask, doe s the system function to bring 
people into care in a timely manner? Do women come to the emer- 
gency room to deliver the child, or do they come in the first trimester 
for some kind of care ? And to stretch our imagination perhaps beyond 
the limits we want to accept at all, does the health system function to 



do anything about the conditions of the environment, the pathogenic 
conditions of the e * --- - v -•* ■ L ’- 



of care? 



Those are different dimensions arid are sometimes included in 

’care.’':'',' •' .■ r 'C" ry Z'Z,.r : 






W BREM: I think we have to single, out different levels of 
evaluation, and there are!. severa!l; George Miller asked, "Should we 
really engage presumably knowledgeable, intelligent lay people to help 
ZZ us ? "jArid Hugh Butt I think said they wouldn't be very helpful in telling 
us whether an internist is really competent to take care of patients 
arid be certified by the vAmericari Board of Internab Medicine. I am not 
really sure that this is where the knowledgeable and intelligent lay 

be in evaluation. The Advisory Board of the American 
Specialties arid the A. M. Ai^Co^cil'on Medical Education, and the : ? 
f fpsLr. cus sRm|7 "off veduca--; r 

/Tiony in which knowledgeable, intelligent lay people would be involved 
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and would determine or influence at least the broad objectives of 
medicine, medical education and medical practice. Perhaps this is 
the level at which they should be involved, and not really in the assess- 
ment of capabilities of individual medical care . Eventually, if programs 
develop to include the kinds of things that Dr. Elling is speaking of, 
then this kind of attention to sociological problems will evolve. But it 
is inconceivable to me the American Board of Internal Medicine should 
have on it as one of its examiners a sociologist. I would hope that 
eventually all of the examiners would be a little bit of a sociologist at 
least, and perhaps even have a little bit of the minister in him. As a 
matter of fact, we sort of look at this sort of thing in candidates, for 
their feeling and attention to the other problems of the patient. 

DR. BOOHER: It seems to me that your lamentations have been 
largely in the direction of the sociological setting of the individual con- 
cerned. If it is true that this set is already determined before the 
individual is ever exposed to medical education, let alone post-graduate 
education, then someone must serve in the matter of selecting people 
who have the proper attitudes that you wish to bring out. As John Jay 
once said, "Since human beings are never likely to be what they ought 
to be, any scheme that requires something they cannot deliver, is 
abortive and nonsense. " ' / : V /•; v 



DR. ELLING: I appreciate very much the remarks about level. 
I would only suggest that there might be a role for someone outside : 
where the question of the broad disposition of the person in the care 
process is concerned. Not so much at the level, "Are 'the right tech- 
niques being employed arid are the right tests being performed, "but, 
rather, in the transfer from one treatment facility to another, ' "Did 
the information go with the patient? Were there any records that went 
along with the patient?" -- this kind of question, the way the person is 
;tidndled ; .genei^U 

CHAIRMAN HALL: Ori the subject, "Defining Objectives in 
Evaluation, " we will hear from Dr. George Miller, who is Director 
of the Office of Research in Medical Education, Center for the Study 
ofMeriitabE Illinois, College of Medicine. / 



1 'Defining Objectives In Evaluation' 






DRV GEORGE E. , MILLER: In his opening remarks. Jack said 
that he expected when we finished! thisVday and a half rneeting, we would 
go home arid do a more effective jobv ! He said this with sorne confidence. 
LmusLexpress 7 .S 9 mewhaf less _:jpptimisni'iab6ut-tM'SVin^re.spect to -the^;:;.^. 



topic which I have been asked to address myself to, which is objectives. 






We have been talking about objectives for a very long time; in fact, 
there is scarcely a conference on education in medicine that does not 
at, some point have a sermon on objectives. Objectives almost seem 
to have become a kind of new religion among those who are educators 
in medicine, or at very least, an altar at which we worship. But like 
so many other altars at which we worship, this may occur only on 
Sundays. We can go about the rest of the week doing what we want to 
do, having done 'our thesis" in this religion. , , 

Really, if the discussion of objectives is to have any meaning, 
it is not in the sense of ,a sacrament, which is the outward and visible 
manifestation of an inward spiritual grace. It has to be that inward 
and spiritual grace which moves us to behave in a certain fashion all 
the time, and not just when we gather around a table like this. But 
% the fact that the talk of objectives doesn't seem to have influenced our 
, v , behavior as medical educators very profoundly up to now should not 
discourage us. Because more general educators have been struggling 



So let me turn then to five of the basic characteristics, the . 
criteria, against which the definition of objectives should be judged, 
and then try to determine the extent to which we use these criteria, 
or manifest them in the statement of our objectives for education 
programs. i- '.'•V.i.i.-’’: cV :// ■■ ■ 

The first, it seems to me, is that objectives ought to be stated 
in terms of behavior; behavior that the learner is supposed to exhibit, 
not the behavior that the teacher is supposed to go through. How is it 
that a learner is supposed to be different at the conslusion of a learning 
experience from what he was at the beginning of that experience ? Be- 
cause by some definition of learning, if behavior hasn't changed, then 
learning hasn't occurred. We've simply reinforced some of the things 
that we believe in, or found justification for behaving in the way we do. 

'if ^ / $o the first of the characteristics of objectives ought. to be in 

terms , of the learner's behavior, not in terms of the nature of the pro- 

^1*31X11, i m /-I 4 - m j-k 4 - Z m n 1 rk vi i rk M i"k rk n vi r— »■» i /fVk «**Uk *» rk Vk Vk rk • n V» nil r% n O O . 



The second is that the objectives ought to be understandable, 
understandable not just toVthe one who states them, but to his colleagues, 
^ : as well; as to his studentsil llt'iis very interesting^if one: attempts to find 

:: from the . member s , of ; a Department of Medicine ; hcw they interpret 

statement of objectives that a student in the course of his experience 
clerkship in medicine should gain skills in dealing with patient 

• . •• ^ r;>- -v\# vM'x Mj'rt-r*! c >.• ■’ 

sWoblems v i Stateddn;,this wa^ it often has only 

idiosyncratic meaning/ it has a different meaning to every person in 
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the department. And each person then generalizes the assumption 
that everybody else means the same thing, and that the students under- 
stand this in the same way. But until a statement of objectives is 
understandable to all those who are to engage in the teaching and in the 
learning, then it means that they may be aiming for different targets, 
even though they are using the same term. 



Thirdly, are the objectives real? By real, I mean, are we 
intending to do something about them? Dr. Elling has called our 
attention to the fact that some of the aspects of patient care, to which 
we pledge allegiance in our broad institutional goals, become sensitive 
to the needs of patients, for example, may not in fact be real in terms 
of trying to do something about the achievement of these objectives in 
the development of our program How often do we construct learning 
experience s that really are designed to provide students with the oppor- 
tunity, not only to gain this skill of dealing in a sensitive manner with 
individual patients > but providing a setting in which there can be some 
observational analysis of whethe r or not they acquire this skill and 
feed-back to tell them whether they have learned the skill or not? •/; . 



An Objective, to be real, means that we permit ourselves to do 
something about it, and not just state it for purposes of the record. 

As most objectives now seem to be stated, it is as though we can take 
a deep sigh of relief when we ’ve got them written down and put them in 
the drawer and forget them and go on doing what we have been doing. 
So they don’t really meet this criterion of reality. ? ;;v : > 



: : ; •; Fourthly, are our objectives achievable ? Are they ac hievable 
in the time, with the resources and facilities and the personnel that 
are available ? Tt is perfectly all right to say that in the course of a 
two week clerkship we want to help students gain sufficient knowledge 
of dermatology, to practice like a: dermatologist, but it is hot an 5 r- 
achievable goal for most students under most circumstances. So that 
the time, i the facilities arid the resources that are available to us should 
in fact dictate to sorne extent the nature of the objectives as they are 
stated. • ^ i J ^ ^ r •• 



Finally, are the objectives measurable ? Measurable may not 
mean the kind of rigorous measurement that we are accustomed to in 
phys ic s or biochemistry. ^.The state of the art of evaluation, of educa- 
tional outcome, may be at the Same stage that the measure merit of the 
speed of light was 100 years ago^rbut we are beginning, and the ques- 
tion is to what extent do we make an attempt to measure those things 
that we say we are trying to achieve ? 5 Because unless we do in fact 
make some attempt to measure them;l.'then we must depend upon our 
own individual opinion of whether : or not these goals have been achieved. 
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Let; us just look at the levels of education to which this group 
addresses its attention — internship, residency, and continuing educa- 
tion, and see how many of these criteria of the statement of objectives 
are achieved. . ■ 

In the internship, for example, I have yet to come upon an 
institution in which there is a clear statement of how the intern is to 
be different at the conclusion of the year from what he was at the be- 
ginning of, the year, except the clear implication that he will be one 
year older. The objectives are scarcely understandable. Without 
some such statement, ?one can question the reality of what it is we are 
attempting to achieve in that year. A measurement of the educational 
objectives in the internship is rarely accomplished. It is obvious there 
are exceptions to these generalizations that I am uttering, ; but if one 
looks at totality, how often are these things done ? In fact, the only 
definition of behavioral objectives of an internship that I am familiar 
with is the one that was established by the National Board of Medical 
Examiners as; part of its study to revise Part III of that examination, 
a critical incidence study, that did define nine major behaviors and a 
number of sub-categories of behavior against which internship might 
be judged, but the frequency with which that is actually incorporated 
into program planning and evaluation at the individual institutional 
. level seems to be : remarkably rare. '■■■■■;/', v.7, 

If we look at a residency program, to what extent are objectives 
spelled out? In fact, in the requirements for residency training, as 
stated in the Journal of the American Medical Association description -- 
the internship and residency brochure — almost the first requirement 
is that the trainee will go through a fixed and specified period of train- 
ing,; or specified set of learning experiences. None of this relates to 
his behavior or the rest.bf these elements of the objectives. 



• 3 The, American Board of Orthopedic Surgery has really made a 
substantial effort ,tq go beyond this in the definition of objectives. 

Other Boards are also directing attention to this, but the time has cer- 
tainly long since passed when we need to define much more clearly 
vwhatdt is we, are attempting to :dq,.i-.;^; 



When we come to > continuing education, I can only say that it is 
: a morass?!, j a swamp. . : ; Almost no one really seems to have a clear notion 

;,of the goals of a continuing education program in terms of the behavior 
of. the learner,, ^except that many of the programs indicate thdt the de - 
sign .calls for the achieveme nt of making the learner better informe d 
about something. So the real question then is whether they have , , v 
bee ome , informe d, or, simply have been exposed to information. 
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I can only say that in the course of a systematic observational 
study which we have now underway, trying to look specifically at the 
way in which objectives are defined, programs are planned; evaluation 
is carried out in continuing education programs, in six different settings 
in a major regional medical program, one thus far can say we have yet 
to find a program in which objectives of the continuing education efforts 
have been defined. Programs are generally developed in an impres- 
sionistic fashion. They are mostly lectures, and the tools are used 
with mixed skills, despite the Council on Medical Education implica- 
tions for twenty years of what constitutes a satisfactory continuing 
education program. ‘ vv-.-;- ;v. •• >v - ;v.v' 

Let me close by turning back to the remark with which I started, 
that is, referring to Jack Hall's comment that we ought to leave here 
and do something. Talk is cheap. It seems to me that if we are in 
fact to improve our programs of graduate and continuing education, 
one of the places that we must begin is that of working diligently in 
the specification Of objectives and not simply believing that if we create 
a program in which learners are exposed to teachers it will be a healthy 
and worthwhile experience. > -J v :y&w:: .ror’o 



Li.. . CHAIRMAN HALL: George, in your description of behavior, 
isn't it necessary to go back to operational definitions, rather than 
traditional' definitions ? It seems to me that medicine has been kind 
of - reluctant to accept; these • ri f .\ • 



DR. MILLER: ; T am not sure that I understand what you mean 
in referring to operational deiinitions,i Jackie .:n.- - 

? CHAIRMAN HALL: 4 We give a description of behavior, or a 

description of what we want, ; and we say "by our decree then; .this is 
the definition of what our objective is. ! 1 It seems to me that we want 
to go back and say, "Hippocrates said that and it has gained support 
over the years, and thus it is the truth, and that is what we are trying 
to achieve. We are going .‘to -have -'to set up our own operational 
definitions in the behavior that we want to achieve ino & : s 

u- DRv MILLER: Indeed, we must, because what we use, it seems 
tot mev^are very loosey definitions . We say that i we want- to turn out a ; : 
general /practitioner but don 't define what are the components of the 
competence- of a- general practitioner. How do you know; one when you 
see one? What it is that he is able to do that people who do not have 
this - competencet are uiiable. to do ? : I use this ; as ran illustration but it 
might-be applied equally well to any of the other. specialties;^ It is-0 v 
equally to be applied to the: continuing education. prbgrams.fpr anyikind-..: 
of" specialistvj% '(■ Miff m >w: tec® 
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DR. WINSTON MILLER: In regional medical programs, we 
are focusing on continuing medical education, and as we try to define 
the objectives we say, "Education for what?,,' ' Then we say, "Educa- 
tion for the role that the individual plays, " and then we say, "What is 
the role?" ’ y? ;vL 4 ‘/.k; :< -...[r". . 

Now; the American College of Physicians gave a: self-evaluation 
examination sometime ago. I took it. It was written for nine different 
specialties, by nine different groups of sub -specialists. It did teach 
one thing, I think, to almost everybody who took it, and that is, humility 
It was absolutely impossible for any general internist to know all that 
information .> a'-V ,?v • : : : *•; V .-.s'* \V. V' 

'! I wonder if you, or Dr. Elling, might? comment about the •. ? 
extremely complex problem we face today in defining roles in order 
to be able to define objective s ? 



DR;V GEORGE MILLER: I would comment by saying that we 
often try to start in areas of disagreement rather than in the areas of 
agreement, which seems to be a fruitless way to begin. There cer- 
tainly are components of professional competence on which we would 
agree;; We ought to be able, to elicit information from a patient. This 
is a skill that can be defined in terms of its operational components. 
We can create educational programs specifically designed to help stu- 
dents gain such skills and can design assessment instruments. It is 
when we get to the broader things when we want pe.ople to become 
sensitive to the needs of others, ?:.that we may bog down. ;r,No two people 
can agree on what this manifestation of sensitivity is. But there are 
many areas on which: we could achieve agreement and that would be a 
reasonable place to starter moving/from there to the more complex, 
rather than being immobilized because we can’t dothe more: complex 
things first.K ^ 












f DR;c ELLING : I just might ask whether there is room for con- 

sidering what some of the c omm on .behavior s are, that are applicable 
across the board, in terms of the diffei'ent specialties. It strikes me 
that .there is.roornforrrecdnsideration of “the 1 now old-fashioned and 
:Outmoded notion of :the;united ebre ofiiknowledge and information and 
skills: that may be? applicable: across^he board;? /Particularly in this x 
area: Of attention! of !patient needs:>and problems and concernsyfea«>x 

MICHAEL:;? ; Areiiweytalking: about^two differentcareas;?;w ; ;'x 
One,? there seemsjto be ^some discussion in evaluatingithe content, 
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this' i? where Dr. Elling’s earlier suggestion of the consumer or the 
disinterested person is being involved. '-C'-v •; '%i. 








Let's take a very mundane example. A patient with pneumo- 
coniosis pneumonia, you say the word penicillin and they get well, but 
the degree of satisfaction of the recipient of penicillin is what makes 
some difference. I am not sure this cap be changed. I am not so sure 
the behavior of the guy who gave the penicillin and took care of the 
patiept would change. ‘ : ’ ■ 

DR. GEORGE MILLER: I do. People learn their way through 
life. We are constantly learning new attitudes, as well as new infor- 
mation; in fact, a whole nation in 1933 learned a new set of attitudes 
that almost destroyed the world. There is no question in my mind that 
we continue to learn attitudes, but in our formal educational programs, 
in medicine at least, attitudes are learned by accident rather than plan. 
It seems to me the time has come for us to direct attention specifically 
to this aititudinal learning and create the settings in which those atti- 
tudes we want to encourage are in fact encouraged, supported and 
.rewarded. : : v ,• 

DR. BUTT: I would like to speak to something Dr. Elling Said 
about a common denominator that we could look at. Even though we 
are subdivided into specialties, is there something that is common to 
all ? In my experience one of the great weaknesses we see in people 
developing in medicine, internal medicine, is a lack of skill really in 
communicating with the patient -- history^taking is the, old-fashioned 
term. This is always assumed to be something you pick up very 
easily but this is a most important skill. ; iTsTV. , v,' \ 

. : : ; ; In. answer, tp the question that George Miller referred to, in 
order to know the person, in order to predict what is going to happen, 
find out what he is going to take home, and this really goes across 
pediatric s , surgery, eve rything else. I think this is a skill that could 
be taught. At the present time there is not any concerted effort to 
teach this. It is kind of obtained by osmosis, by listening to someone 
» v .->else.-: : .; v -^ 7 ,.?; : ■ ' . V.Viv .,••• Ink ' ; ' 

•* . • DR. BREM: I question whether this sort of thing can be taught. 

I think Dr.i George Miller touched on ap important thing -- you can 
/ -create an environment* an environment that consists principally of 
v ; people, ; in the way of preceptors, and it is. the way they manage them- 
selves that determines what the students are going to do. Really, it 
is our environment in the medical schools over the year’s that has 
. : fallen short. But I think we are getting a little more insight. I thipk 
this is the only way that these attitudes can be developed, .. 

; Z'kk'vi - i::; DR. -GEORGE MILLER : And I think you illustrate by this 
:i; : comment, Tom, one of the problems.; When. you, think of teaching as 
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something to do in courses, teaching is the total environment that we 
create in which students learn. 



DR. ANGELIDES : How many medical schools have gone through 
and made a simple analysis of what the third year student needs ? I 
would submit there isn't a medical school that has done it. There are 
really four things you can teach him that he needs to know for the rest 
of his career. They are so simple that it is almost trite to bring them 
up. ‘ ,v'.- 



One is history taking. You have to teach the student to hold the 



patient's hand and listen to him. There are the techniques of physical 



examination. When you 'ye: accomplished this, you intr oduce differen- 
tial diagnosis. How do you organize thought processes, interweave 
them, so you can come up with a diagnosis ? Lastly, we must instill 
in the student the desire to ask himself, "if I come to this conclusion, 
how can I explain it? " or in our parlance, think patho -physiologically. 



We do this every day and have tutorial instruction to do this. 



DR. FREYMANN: I would like to ask about the total environment 
as the place where we learn. 



I have seen one article that says that, written by a third year 
medical student in the New Physician . Has anyone else said that in 
medical journals? I'm sure the educational journals are full of it. 



CHAIRMAN HALL: /You might go ahead and say the rest of what 
he said in his article. h " ! \V ' :• 



DR. : FREYMANN: Your environment is what teaches you; it is 
not what your teache rs teach you. : /./ : . '' '/> 0 • // \ ///•' . . /•/.' ■ .• ' 



DR. DAVIS: As you probably know better than I, evaluation is 
a process in itself and really only became respectable after World 
War II. Initially, it was applied after systems, activities or processes 
were started and decisions had to be made as to whether to continue 
them . E valuati on ap plie d to a process like education turned out not to 
be successful when c bns ide red as an afterthought. Evaluation is effec- 
tive if applied as an integral part of the process, from the very beginning. 
And applying it as an integral part of the process means that it has to 
be done by people who are as competent as the individual who is framing 
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: Now",^ ^ as nearly history; can show, ' evMuation has, : 'in general, 

not been effective in changing the course of a system when the evaluation 
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was started after that system was/in being. . Therefore, / I was curious 
whether you thought that maybe some of us were a little too impatient 
in trying to evaluate a process, such as medical education, which has 
been so long without it. U y< C-J. '■ • /- ,;v-_ •; v'c- =:• ; • 

DR. GEORGE MILLER: I would endorse your view that evalua- 
tion is part of a dynamic process. I would not endorse your view that 
we are too impatient. I think we aren^t impatient enough, because if 
we were impatient enough, we would have gotten along with the business 
of incorporating it into the dynamic process of education, and it is time 
that we did. 

CHAIRMAN HALL: Dr. Hubbard is here to address himself to 
"C ontemporary Activity in Program Evaluation. " 

Dr. Hubbard is President and Director of the National Board 
of Medical Examiners and leader in that field for several years. 

"Contemporary Activity In Program Evaluation 11 

DR. JOHN P. HUBBARD: You have asked me to speak about 
contemporary activity in program evaluation. This could be a very 
broad title. , I am. going to limit it, if I may, to the activities of the 
National Board, since this is something I know about. 

. The National Board became involved in program evaluation 
when we changed from the time-honored essay form of examination 
to the more reliable type of multiple choice testing. The historic role 
of the National Board, is to examine the individual in order that he 
may attain qualification for the practice of medicine.; This went on in 
the early days of the National Board and still goes. on,. , but when we 
Changed to the multiple choice testing, medical schools became very 
much interested in our. examinations for impartial^ examination of an 
entire class, usually the second year class, or the fourth year class. 

It soon became apparent that when a whole class takes the. same 
examination; -- an’ extramural, /^objective, reliable examination - - the 
school obtains a good deal of information about the process of the educa- 
tional system, as well as the product of the educational syste/m. :. This 
still goes on extensively; at . the undergraduate leveL; : ; But I think this 
evening we are more concerned with the graduate level, and I will 
Conftne/my;-remhrks.itb;,this ariea^ 

/ ; The. study and revision of our Part III examination led to the 

current form of , that examinatioh./V have described it as "Objective 








Measures of Clinical Competence. " This is not meant to stand alone 
but to stand as the third part of our series of examinations, assuming 
that we have already tested, in Parts I and II, as comprehensively as 
we can, the knowledge of basic sciences and clinical sciences. 

Our new methods -- that we have described as program testing 
coupled together with our more traditional multiple choice techniques, 
have brought and continue to bring in requests from specialty boards, 
specialty associations and fr om many directors of medical education 
in hospitals for help in the evaluation of their candidates and trainees. 

The specialty boards with which we are now cooperating in the 
development of their certifying examination are the American Boards 
of Anesthesiology, Gastroenterology, Internal Medicine, Ophthal- 
mology, Pediatrics, Plastic Surgery, Psychiatry and Neurology, 
Radiology and Surgery. In-training examinations for residents have 
been undertaken for the American Board of Neurological Surgery and 
the American College of Obstetrics and Gynecology. Self-assessment 
programs, similar to that of the American College of Physicians are 
now underway on a cooperative basis with the American Academy of 
Pediatrics, the American Psychiatric Association, and the American 
Society of Anesthesiologists. 

The in-training examination for the Board of Neurological 
Surgery deserves special comment. Their concern was the high per 
cent of failures that occurred at the time of the certifying examination. 
After seven, eight, nine, or ten years of training/ 45 per cent of their 
candidates failed the examination .: 1 What was the matter ? It seemed 
very wasteful. Was the training program at fault? Were the candi- 
dates perhaps inadequately selected to begin with? Was the examina- 
tion at fault ? The Board of Neurological Surgery with our assistance 
set up an examination to be given as an in-training examination for 
residents with the firm understanding on the part of everybody -- the 
individual, the program director, the specialty board -- that the 
results of this examination would in no way be fed into the certification 
procedure. The grades would be reported only to the trainee and to 
his program director. In this way, both the trainee and his program, 
director could take note of weaknesses and would have the opportunity 
during the remainder of his training program to remedy these weak- 
nesses. This has been continuing now for some five years, and during 
this period of time it has become very apparent that the program is 
yielding considerable information! about the training program itself, 
as well as the ./.calibre' of the individual#^ ' ; 5 =. ' : 

■ : 3 , in 1964 we published a paper entitled The Internship: An 

Evaluation of Input and Output.¥ This brought a number of requests 
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from hospitals for help with evaluation of their training programs. 9- 
One of the fir st requests was from Dr. Jack Hall, who wanted our : ■ 
assistance in evaluating the internship at the Methodist Hospital in 
Indianapolis. We accepted his request and set up a pre-test and 
post-test evaluation of the internship year. At the same time, we 
undertook a similar pre-test, Vpost-test evaluation' for the military 
hospitals in the army, ' air force; and navy, at their request. It soon 
became apparent that there was limited value in this kind of evaluation 
for individual hospitals, j The number of interns in any one hospital ' 
were too few, the training programs are too variable. We have there- 
fore limited our participation in evaluating educational programs at ;1 
the graduate level to those that have sponsorship of some kind on a 
national basis: a specialty board, a specialty organization, or an 
association in thevfieldsof graduate , and continuing education. ■"?: 






At this point I would like to offer a suggestion. Perhaps this 
might be better called a challenge for the Association for Hospital 
Medical Education. I don't have to remind you that the heat is on with 
regard to the problem of the fore ign medical graduates . ; ; 'I ; am sure ? 
that one of your very earnest concerns, the concerns of all directors 
of hospital education, is the competence of the foreign graduate, serv- 
ing as intern or resident in our hospitals. There are reliable ways to 
measure this competence, at least certain aspects of it, if — and I 
underline this word "if "i-- if a well-defined program of evaluation were 
to be seriously undertaken on a national basis. What better answer to 
this question than a program undertaken by and designed by the Associ- 
ation for Hospital Medical Education? I have in mind the program that 
was undertaken in New Jersey. Maybe you are familiar with it. Under 
the New : J ersey Hospital Ass ociation, with the leadership of Dr. Erwin 
Hirsch, a program was set up for evaluation of the internship programs 
in that state. There was a ve ry obvious motive , and that is, the very 
large percentage of foreign graduates in the New Jersey hospitals. 

This was a bold and forthright kind of undertaking 9 I leave you with 
this comment -- perhaps what was good for New Jersey might be good 
for the nation. 






DR. NUNE MAKER: Y ou may have been a ske d alre ady ab out 
osteopaths. / Last September somebody suggested a national evaluation. 
Have you had any experience up to now with -any osteopaths 

•; viDRi HUBBARD: Not directly. 7 - We have:indirectly,\; since a— 
number of' states turn to us for help in the examinations; ; and we pro- ; ' 



vided states with our Part I and Part II examinations, or whatever :i - ; 
they may have asked for. Certain of these states have joint boards 
for M . D . ' s and oste opath s . So the re are osteopaths who take our 
examination under the auspices of the state examining board, but we 
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don't have data from this. These states are very careful to give us 
identification by number only. G ertain state s that have combined ; 
boards are now using the new FLEX examination which is a step f 
forward, in our opinion. , -;,v ; . --'Ev < •V/'-"?'. - ■ (-■ vr-' ! 

DR. NUNE MAKER : Several years ago we were very much 
interested in co-sponsoring this sort of thing, but ran into a budgetary 
problem. But we were interested spectators .in the New Jersey experi- 
ment. The question I would ask now is, was that large enough a sample 
so that you would learn more nationally than you would learn from the 
New Jersey experience fcki ;£v '.'v-v -iv -'K ; ' ■' iv, 

DR. HUBBARD: The New Jersey experience for the most part 
yielded information with, regard to group comparisons, foreign gradu- 
ates vs. U. S. graduates^ years out of medical school, things of this 

sort'./ - iy U if- <•;. ; ./ 'y •■%&*.?; "■'<> t i V ; • i. - " •/ ■: - c\ 

It is a very important point to keep in mind — if any such ;; 
program should develop — that there would be limited evaluation for 
a hospital that had two, three or four interns. -.-.Ilf over a period of V 
years a hospital turned up with very poor performance repetitively, 
there would be considerable meaning in this, ffe :v ^ ^ 

CHAIRMAN HALL: Dr. Hubbard, could you elaborate a little 
bit more on the study conducted in New Jersey?. . >• ~ • 

DR. HUBBARD: This was a pre-test and post-test activity. 

The Part II examination had been taken by those who were U. S. gradu- 
ate s £ We admini s ter ed Part; III at the be ginning and again at the end 
of internship. Comparison could then be made between the pre- 
internship performance, and the post -internship performance. 

, ; DR. FREYMANN: I would like to know more about the critical 
incident procedure'^^'-,^^^ 

DR. HUBBARD: The critical incident procedure was developed 
about 1958 or 1959. « We were concerned with the Part in examination. 
This, was the traditional orali*bedside type of examination. As we . i i. / 
became more familiar with the high degree of reliability we had in 
our Part I and Part II, we became uncomfortable with the lack of 
reliability in the oral procedure. . There were too many variables. 

The .patient :that; the candidate examined wasv one variable;;?;: The 
examiner was a variable . And another variable was the candidates ; ; ; 
we wer^^trying tb/meas^ev 
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We undertook a grant supported study that proceeded over a 
couple of years. We sought the help of the American Institute of 
Research and' its Director, Dr. Flannigan, who is the originator of 



the critical incident technique., The question was: What is the behavior 
of the intern that we,, the National Board, are trying to measure in our 
Part III examination? "y" . 



Those directly in charge of interns were asked to cite "critical 
incidents," that is to say what an intern had been observed to do that 
was particularly good or particularly bad. This gave us a br oad spec - 
trum of the abilities of the intern: his skill in taking a history and 
doing a physical examination; his judgment in ordering s.nd interpreting 
diagnostic procedures and his judgment in therapy and management of 
patients. Then we! got into some of the more difficult areas to measure, 
such as the physician's responsibility for his patient, his relationship 
with his peers, his relationship ..with hospital authorities.. We accepted 
our limitations in measuring some of these latter categories of behav- 
ior and concentrated on those for which we felt that we could obtain 
reliable assessments. 



We', introduced the motion picture to standardize the patients 
that the cancdidates would see; we standardized the measurement of his 
judgment 'in: taking care of patients through the introduction of the now 
familiar ’'erasure technique " for patient management problems. 

MEMBER: Did most all the American graduates in New Jersey 
who took the pre-test after Part III pass the examination? If so, does 
this suggest maybe they didn't need the internship ? Or weren't you 
measuring something that happened in the internship ? If this is true, 

:• . what is the implication? " y"y , 

DR. HUBBARD: Most of those who took the examination did 
pass,. Only about two per cent of American graduates, after internship, 
fail on Part III. There were more of the foreign graduates that failed, 
and their mean performance was significantly less good. "" , ■ , 

v, .. r We are continually studying the validation of these measurements 
We have administered the Part III examination to students in the third 
year, the fourth year,' the internship, and, now we are getting into the 
residency area. There is a significant increase in the grades between 
the third year and the internship, s o I think something" is happening in 
the fourth year and internship that we are /measuring with our Part III. 

y'ij"" "y MEMBER: The problem we keep running into is the impact of 
a repeated test in the training program. The candidates being tested 
are being prepared and respond to the test in verbal combat and 
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regurgitate perhaps inner information in response to the test, rather 
than maturing in -a broader sense. 

Can you comment on that in reference to your last statement ? 

DR. HUBBARD: I find it difficult to do so in a brief moment. 
We are providing an answer to students who want to know the answer 
to the question, "How am I doing?" This is criticized in some areas. 
I think there are obvious values that can be gained on the part of the 
student and faculty in accurate appraisal of both the product and the 
process of medical education. 

CHAIRMAN HALL: Our next speaker is Frank Husted, who 
is going to address himself to the subject, "Judgment Evaluation. " 
Dr. Husted is Associate Dean, 

State 



'Judgment Evaluation' 



DR. FRANK L. HUSTED: The subject that Jack Hall gave me 
to talk about tonight is a rather difficult one and yet is one that I think 
deserves some very serious consideration. Also, it is probablv . 
tautological, but then I do think words have a way of irripe di/ag c om- 
munications at times, even though it should not be the case.; At other 
times, even misusing them serves a purpose. ' /' 

Dr. Miller, in his usual efficient and effective manner has 
most admirably outlined the bogeyman of education, the teaching- 
learning objectives. Inherent in his comments is the element of j 
definition. I would like to enlarge on that component by directing 
your attention to the role of definition in judgment evaluation. Dr. 

Hall 1 s letter directed me to introduce and discus s techniques for 
evaluating judgment; to elaborate on the "state of the art, " as it were, 
with the implicit Objective of passing on to you a bag of ready-made 
tools or instruments with which such evaluation processes can be 
effected. Jack knows me well enough to appreciate that T do not always 
follow instructions to the letter. -’I hope ttiatTn positing a broader base, 
I can introduce a pragmatic foundation from which each of you can 
better assess techniques, or, even better, provide a baseline from 
which you can more effectively develop your own instruments in terms 
of your objectives -- your criteria ^- to meet your special hee^ds. / 



— Benjamin BlOom,^in hisf"l 
Miller/' ih’ an article which appeared in the. March 19frl ; ;'issue of the 
Journal of Medical Education; Abrahamson, McGuire, Ros inski, 1 v ‘ 
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Thomson, and others skilled in evaluation state explicitly or ! by infer- 
ence that evaluation must begin with a clear definition of behavioral 
objectives. I would add to that for present emphasis that judgment 
evaluation must begin with clear definitions of what it is that one is 
evaluating. One cannot talk about, judgment qua judgment;; one must 
talk about judgment in respect to something . Clinical judgment ? 

Still much too broad! For surely the ability to "judge" and by such \ 
"judgment" rate a heart murmur as Grade I, or II./. or V is quite ii 
different from "judging" the condition of diseased tissue and from 
such "judgment" rendering a definitive diagnosis. And these are 
dramatically; different from the judgment which follows each of them 
and which guides or determines patient management. 

Can I compound the issue and confuse you further by suggesting 
that we are considering basic issues involved in judging a person's 
ability to judge -- to exercise good judgment? Let me submit that we 
are observing the behavior of a person or persons engaged in activity 
which requires a , synthesis and effectuation ,of knowledge, and which 
requires skills, attitudes and understandings contained in behavioral 
objectives. These are, or should have been, explicit components of 
the teaching-learning experience through which the observed has 
recently passed or in which he is currently involved. 

Let me add further, if you expect someone to hand you a ready- 
made "judgment micrometer" or a 9/16ths "judgment wrench" to 
measure or slip onto a 9/16ths "judgment laden nut" then I submit that 
your expectations will go unsatisfied. For in accepting such an instru- 
ment you have made an apriori judgment about its efficacy, avoided 
the definition process, and you will draw inferences from the results 
which may well lead you to throwing out the flowers with the weeds. 

Let's move ahead, then, by going back. Back to the basic 
premise of definition. Whether you elect to use an instrument designed 
by experts or elect to construct’ your own, you can do so only by clearly 
defining the behavioral components and the criteria with which you will 
draw conclusions re: judgment. ; It is wrong, it is dangerous,, and it 
is saying a risky nothing to say that "he displayed good judgment/, " 
That's only half the loaf and in this case it is not better than none. 

Some define judgment as the process of forming an opinion. 

Okay, let's look at that for a- moment. That is a behavioral activity, 

; isn't it ? But something is missing if we are going to use it in an 
operational scheme. ; , Jack referred a few moments ago to operational 
objectives. We must ask the crucial follow-up question: "About what?" 
Haying answered that, we are still in alien territory.' We have to de- 
fine the. "what" in specific behavioral terms and then set values of 

'/••t '• , : ;-v : "''.s' 







■ ' ' • ' ■ 












y ; ,t 




•••;. 'Cr.: 





a- • . . ‘ . 

: -n. .. 



■JL 



\ 



"good" or "poor, " "go" or "no-go" on the behavior or behavioral 
matrix. Even then we must validate our scheme by having other 
judges render decisions, or enter into the process of forming an 
opinion, and thereby arrive at some estimate of concordance. The 
crucial question then becomes judgment about what? Or, better, 
judgment in respect to what? 

In the present context, I am not happy with the definition of 
judgment I just- gave you. I want to push it into an action frame of 
reference and suggest that the judgment process of a health profes- 
sional consists of a series of many judgment evaluations, each the 
product of instruments developed specifically for the behavior one 
wishes to assess. Having measured the knowledge and skills with 
the appropriate tools; observation -- with rating devices -- might 
well be developed. With these tools the elements identified in the 
definitions of behavior previously developed can be charted and coded. 

Thus, in observing the behavior of an intern or resident as he 
examines, diagnoses, and sets up managemerit processes for a cardiac 
patient, one can evaluate the judgment he displayed by referring to the 
total value of the discreet ob ser vat ions made in proce ss, rather than 
making a sweeping statement relative to judgment. Having evaluated 
the student's judgment in this area, is it safe to assume that he will 
use equally good judgment in the examination, diagnosis, and treatment 
of a patient with a gas tr o-irite stinal disease ? No? This then requires 
different knowledge ? Are there, then, both general arid specific ele- 
ments in judgment ? If so, then both "general' 1 and "specific" definitions 
and devices need to be developed, and, I might add; general and specific 
criteria. Do we not then need to carefully define the behavior for each 
situation, at least in broad categories ? How, then, can we speak of 
judgment evaluation without being vitally if not anxiously concerned 
about definitions ? 






How many times have you saidj "l am going to have to really 
think about it. It is a very important question. " How many times 
have you said , "He displayed or used good or poor judgment in handling 
that patient. " What did you mean? Did you examine at the time exactly 
what you were referring to when you made the statement about that 
intern or resident, or student? When you said that he used good judg- 
ment or bad judgment ? v'- :: : 

If asked, you would begin to describe the behavior of the student 
or the resident. You would cite critical incidents or give anecdotal 
evidence to support your judgment, but you would do this only when 
someone asked you specifically, "What do you mean? " I see a few 
lights going on.y~ I have a sneaking suspicion that the beliavi oral indices 
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are known,, that the, criteria are defined,, the elements of a measuring 
instrument are lurking in the recesses of your mind. It just takes an 
incident to bring them out,, someone to question you, someone to pick 
at you for a moment and ask you, "What do you mean, and about what 
do you mean it?" Judgment^ evaluation cannot be left -to that kind of 
post facto functioning.;, V' .. .f 

. , That this subject was placed on the agenda of this meeting is 
evidence of the fact that you accept this premise,. Good !, We agree on 
that. Now, then, let me suggest that a recounting by me or anyone, 
else of available shelf-type instruments will,; at best, be an exercise 
in futility, unless, unless you can say with confidence I have defined 
the behavioral components, I have established the criteria, I do have 
a series of incidences with which I can observe the behavior of the 
student as he goes through the process of forming an opinion. And if 
you've done all of that,; you don't need Dr. Husted's "sure-fire snake 
oil" guaranteed to purge the academic system and render valid judg- 
ments of judgment.;,;;, *».>:*>: sfi ** Hi' 

y j'yyVr; May I summarize ? Judgment evaluation, as any other evalua- 
tion system, rests squarely on the effectively developed, predetermined 
rationally derived bases, of:,y,vy' ;• :v.y 

The definition of objectives in behavioral terms. , 

i / The establishment of criteria.,;. Standards of acceptance ' 

through critical incident and act-oriented indices. 



scale, 
sheets. 



-The implementation and continuous critical appraisal of 
. ; v . : ,, each use to. which it is put. , y. y • 

And these in turn are based on tw r o basic assumptions: : 

1. There are general behavior patterns which are common 
;;;y ,,-, ; t° all situations requiring '.'judgment.;,"; y-' Vy ; ■ V 

There, are specific behavior patterns indigenous; to they. ~ 
specialized body on knowledge upon which sound judgment 

\ yv-y..- is. predicated, y uyri J : .? TiY iym ; v- 

vyi;(y,vfPR.', ? GREGORY : Dry. Husted, what would ypCi do in this circum - 
stance : You are examining someone with an immediate problem and 
you make adjudgment, about his judgment and conclude. that it is a fine 
judgment. But some days or weeks later it ends in a total disaster . 
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And in relationship to that; what is the role of intuition, and is 
intuitive thinking permissible in any way at all ? Has it. any value, and 
is it in fact just a judgment process that no one can properly define ? 

DE. HUSTED: Let me suggest that if you have developed the 
behavioral indices, and if you have carefully defined what it is that 
you are looking for, asking for, and searching for in this judgmental 
process, then you will be less likely to make that kind of commitment 
about whether this person did or did not have good judgment. However, 
there is always in any measurement system what is known as a margin 
bf error. I really don't care what system it is, there are margins of 
error. I don't think we will ever get instruments so fine that we will 
hit everything totally . ‘ f : ^ ; *'■ 



Concerning your second question, I would have a sneaking sus- 
picion that in this area of educational evaluation, particularly in 
judgment areas or where our tools are relatively weak, that we will 
never be able to eliminate intuition, nor would we want to eliminate 
intuitive thought. Intuitive thought, when examined, may not be as 
intuitive as imagined. It is predicated on some pretty well established 
ideas that you have. It appears to be intuition simply because you 
haven't examined the basis on which this intuitive judgment is made, 
or the intuitive conclusion which is reached. I have a suspicion we 
just have to look at that which produced it. ; V 

DR. WINSTON MILLER: Is high level judgment teachable ? 

DR. HUSTED: I really don't think I have an answer to that 
question. Anything for which you can define behavior is amenable + o 
a teaching-learning situation. I would rather that the question be 
asked, is it learnable? And is it learnable in the same sense that 
attitudes hre learnable ? Yes, I do think it is learnable, but I think 
there are many problems, if you are looking for a specific structure 



or a specific teaching-learning situation where you say, "I'm going 
to teach judgment. " . Like attitudes, it has to be something that is 
introduced in process. 

DR. KING: Doesn't judgment have to be based in knowledge of 
what you are going to do, first of all? You can't teach judgment to an 
individual who doesn't know much about what he is going to judge. 

DR. HUSTED: I submit the individual can't learn judgment ' : 
about something about which he is not knowledgeable. We consider 
the behavioral objectives referred to by Dr. Miller include knowledge, 
and include attitude, and these are part and parcel of the educational 
rinntn a& I don't'think you can teach clinical judgment, if you will, to 
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an automobile mechanic, no matter how clever he is, unless he has 
some knowledge of medical content around which he is asked to make 
a judgment. .;••• • > >-•. j-: • , 

DR. KING: Judgment fundamentally is based on knowledge, 
isn't it? It is knowledge on which a man can make a judgment, some- 
thing a non-knowledgeable man can't make a judgment on. 

DR. HUSTED: I wouldn't want to get into the trap, that because 
he has the knowledge he is thereby able to make good judgment. 

DR. ROSINSKI: I think the point was raised where one was 
predicated on the other, and I think it is a good point. If you are going 
to make a judgment about the use of two drugs, you are going to have 
to know something about the drugs. - • 

! ■ V: DR. HUSTED : Yes, but the fact that you know something about 
them does not mean you are going to make good judgment. 

DR. ROSINSKI: Of course not. 

DR;. HUSTED: I agree, the judgment is predicated on knowledge, 
but the presence of knowledge does not necessarily mean that good judg- 
ment will be effected. >• /. ■■■-' h ■'•}: v.|r.' •* j-.- -.y'-i uq-. 

DR. .BREM: I was going to raise exactly that issue, that a man's 
judgment-is no better than his knowledge ancV' it can't; be. 

. ; . - , DR. KING: Isn't their clinical judgment equated with their 
decision-making capability, based on incomplete or indecisive , 
information? /.■: vf'V/r'A '- }:■■■"■; ‘ •. ' 

DR. HUSTED: All of which is evident on the basis of observa- 
tions by the judges, and these observations have ; to be predicated on 
prior kinds of base lines. 

DR. EVANS: There are general knowledges that you can teach 
about judgment decision-making which will then lead you to know the 
necessity for going and finding out about the two drugs, so, that you 
, can make the decision, vl;- \ ; /.>•■>• ;V..- 

HUBBARD: Can we use a simile here in our Part II and 
v : Part III examination? Part II is. basically testing knowledge and the 
^individual's ability to apply his knowledge to the problem with which 
he is confronted. Part IH has an element of testing judgment. Now, 
we have seen individuals who display lack of judgment, as measured 
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by our Part III. Having passed Part II, they have knowledge, but they 
can’t altogether use the process of judgment in meeting the problem 
before them. But we would not see an individual pass Part III who had 
lack of knowledge to begin with. 

DR. HUSTED: I would have some question as to whether or not 
Part III measured judgment. It probably does in some situations, but 
it is just a manufactured situation. It is far better that it be done by 
some more regulated observational approach. ^ ■ 

DR. EVANS: Have you ever given Part III to anyone who failed 
Part II? Do you actually know who passed Part III without having passed 
Part II? '■■.■ = ; : ‘ : :v 

DR. HUBBARD: The answer to your first question, yes. We 
have an experiment in which Part III is given to medical students. I 
still will hold that in Part III, we are measuring something Other than 
we have tested out in Part II. 

DR. MICHAEL: I would like to direct a question to Dr. Gregory. 
It seems in your orthopaedic examination last year, and the year before, 
the last group of questions that had to do with clinical judgment, indeed, 
the scores worsened as the men progressed in their training. There 
was one group that worsened in their factual knowledge. Am I correct 
in that ? 

DR. GREGORY: If I understand you, there was an interesting 
event that occurred on a patient management problem, given to resi- 
dents or given to applicants for certification, and examiners of twenty 
or twenty-five years' experience in the practice of orthopaedic surgery 
demonstrated what I was referring to earlier, a, kind of intuitive sense 
about the case being described and often cut directly through to the 
diagnosis without traversing the traditional and proper pathways. That 
was a remarkable thing that I can't explain. 

DR. McGUINNESS: They have a lot of stuff built into their 
computers that does it for them. v ; • l \:V’ ; 

DR. GEORGE MILLER:. If you stated it correctly, I'm not 
sure that the interpretations I hear around the table I would consent 
to. I think it is a question that requires further study with a larger 
sample/' I am not at all sure that those who leap to conclusions 
necessarily render better patient care. nr : &(•.)■■ • /■ : v , 

DR. MICHAEL: As far as the residents were concerned, year 
by year they were worse;- / =5 v vIV' •: 
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DR. GEORGE MILLER: The residents, year by year, showed 
no increment in the main score of their ability to manage these problems 

DR. MICHAEL: Yet in their recall of knowledge, the re was an 
increment. . ... , >. • •••■.., 

DR. GREGORY: There was one other venture we had and it 
may have had something to do with that, and that is the impact of one 
individual on another in the matter of collective judgment about things. 
We attempted to build this into the oral examination for two or three 
years, and I think perhaps the reason we abandoned it was because 
our examiners had such a devilish time trying to maker any assess- 
ments. But the impact that we observed was quite clear. .There 
usually was one, or two persons -- and these were groups of six, I 
believe but one or two were far more vocal than the rest. , .They 
would tend to dominate the conversation and dominate also the conclu- 
sions that were drawn. Here is an aspect of judgment that I'm sure 
is part of every, day life. You solicit another viewpoint to reaffirm 
your own and if it is sufficiently persuasive, it may change your, mind. 

CHAIRMAN HALL: The hour is late but I think we are address- 
ing ourselves now to an extremely important part of evaluation, and 
that is, "Whose Evaluation? " Dr. Edwin Rosinski, Professor and 
Head of the Department of Health Education Research, University of 
Connecticut Health Center, will address himself to this problem., 

: . "Whose, Evaluation ?,|' 

DR. EDWIN F. ROSINSKI: A summary, is going to. be extremely 
difficult, considering the kind of group we have here today. It will.be 
a summary in the sense that I will interject my own personal feelings 
about a number of the items and questions that were raised, and make 
some specific observations. 

Dr. Elling really set the tone; which is, evaluation. He focused 
: on the real problem, one we haven 't really addressed ourselve s to, and 

that is: defining the universe. \ In this case, the. universe would be, to 
use Dr. Elling' s definition, health care. He pointed out that we- are not 
really sure of what it is we are talking about when we talk about the 
delivery of health care. If we haven't been able to really define that 
universe, then I question whether we can, go ahead and evaluate indivi- 
duals . Because we haven't been able to define that health care universe, 
as a consequence, we evaluate students in the same way we deliver 
medical service. It turns out to be episodic and crisis oriented. ,y s 
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Because the universe has not been defined and because our 
evaluation is episodic and crisis oriented, what we've done is focus 
on the outcomes of education. As soon as we focus on the outcomes 
of education, whatever little block of the educational component we 
select, we make inferences that, the individual is now going to be able 
to do something differently than he did before. I really wonder if that 
is so bad? 

For example, I would be relatively secure in knowing that a 
resident is learning how to do a lumbar puncture. If he is learning 
this, I feel fairly comfortable that he will be able to do it. I have to 
make these inferences and I think this is what we do a great deal of 
in the process of evaluation when we focus on educational outcome. 

We know we can evaluate his acquisition of knowledge about drugs, we 
infer and, hopefully, he is going to make use of this knowledge . 

Many Of us have had a tendency to criticize much of medical 
education because it is focusing on knowledge. I'm beginning to wonder 
if in the present state of the art of evaluation, whether perhaps we may 
have to remain content for the time being just with this measurement 
of knowledge ? 

I think that we will probably be provided with new and significant 
data from the work being done by the National Board. In terms of pro- 
fessional behavior of the individual, is there any difference in results 
from esoteric examinations and those that measure pure knowledge ? 
Because our evaluation process focuses on knowledge I am not going 
to be too shocked and too concerned that we haven't become more 
sophisticated in our evaluative techniques. I am concerned that we 
may get to the point in the process of evaluation where we can become 
s o detailed in our definition of goals and objectives, that we ' re going 
to end up creating, or having to Create, some really esoteric evalua- 
tive techniques to find out whether this is being done. 1 ’ 

We are going to end up like the congregation who got into a 
hassle over whether they should buy a $100 or $10, 000 candelabra. 

The whole congregation was split over this issue, so they finally formed 
'• a committee to solve it. They couldn't solve it either, so they went to 
the president of the congregation and told him what the issue was -- 
that they were split on a $100 or a $10, 000 candelabra. The president 
of this cdrtgregatiorisaid, "if we decide to buy the $ 10, 000 candelabra, 
are'we sure soriiebody cari play it ? " V : : ' 

1 This is what I'm a little bit worried about. When measurement 
techniques become too esoteric, the people, the rank arid file medical 
educators, at the undergraduate and graduate level who are going to 
have to use these, may end up being completely be wildered by them,, 
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In spite of my criticism we are going to, have to move along 



the lines that Dr. ; Miller and Dr. Husted suggested. But, I think more 
than anything else, we are going to, have to face up to our responsibili- 
ties. That really what we want to evaluate is the individual so he can 
go and face the real life as a practitioner,. But to do that, Dr. Elling, 

I think we are going to have to go right back to the very things you 
raised. We are going to have to get at, ultimately, such issues as, 
what is medical care and what is health care? What is quality care? 
We are going to have to address ourselves to the issues of, who can 
best deliver that care. If we get answers to some of these, I think 
then we will be able to define better the individual who is going to be 



responsibility is going to be yours. 

The proof of the pudding is going to be, "How are you going to 
make use of the kinds of suggestions that are made here?" The whole 
process of evaluation is going to be your responsibility. You're going 
to have to define it. I would like to see us get to the stage where we 



medical education. We cannot, because of a lack of knowledge in medi- 
cine, do the impossible or do the job for you. We can’t do it without 
the help, without the stimulation and without the input from you. We 
are in-house consultants, and the educator who knows nothing about 
medicine and who does the job of evaluating, I would suspect is an 
outhouse consultant. 

DR. ANGELIDES: How does one define the Universe? I think 
we are currently evaluating no known objective or no known definition 
and any evaluation criteria we establish are on shifting sand and in 
some cases on quicksand. 



we are striving to do. Once you establish that, you know what you are 



doing all of this. If we define him, then we can evaluate him. 



We don’t have the answers, I suspect. Yet, the ultimate 



can define our product in terms of behavior that is understandable, 
realistic, usable and measurable. As we do that, we can then evalu- 
ate with greater confidence. 



DR. HUSTED: I would like to add one comment to what has just 
been said about the educators and tie it in with a comment made earlier 
The pure educators, like Ed and myself, haye come in the back door in 



I don’t think we ’re ever going to be able to evaluate until some - 
. uuuy attempts to put down in clear, understandable terms, what it is 




evaluating for . Right now we are just evaluating 



CHAIRMAN HALL: I think that we have our work cut out for 
us tomorrow. We have heard the challenge, those of us related to 
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specialties and special areas of interest in education, related to health, 
defining our universes, setting up our criteria, and finding the most 



■ ' • ,'{*/' Y ; : 



I am looking forward to an exciting day tomorrow. : ' 
. v .‘ The meeting recessed at 10:20 o'clock p.m. . - . 
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CHAIRMAN HALL: This morning we would like to explore. the 
possibility of applying the things that we discussed last evening.- We 
have had leadership in this from people like Charles Gregory and the 
American Board of Orthopaedic Surgery, and their experience overt 
the past five years or more with their program is, I think, quite 
meaningful to us, and to our intent of improving the evaluatiori of 
graduate medicine. '■ ■ ' 

.. • ■, ■ .... ; . . •’* ■ 

DR. CHARLES F. GREGORY: First, I wish toiaay that I will 
not embark upon any technical discussion of the examination that we 
have developed in conjunction with Dr. Miller. The phalanx of experts 
would quickly strip away my veneer of expertise and leave me naked. 
So, I should rather tell you about what this has done, as we see it now, 
to our ortthopaedic training program, and what it has made possible, 
we believe, in the way of additional development. 

Some history is significant, I think, because in 1960, *61 and 
*62, the American Board of Orthopaedic Surgery was wrestling with 
a problem that is common to a good many boards: how to account for 
the large number of failures on board examinations from among candi- 
date s who are pre siimably produced by excellent training programs . 

Now, it happened in 1962, in Highland Park near Chicago here, 
that the training chiefs for orthopaedic programs were assembled, and 
addressing them one afternoon was Dr. George Miller. The essence 
of his remarks was we. didn't know what the hell we were doing. A 
perceptive me mbert of the audience , D r. Charles Herndon, who wa s 
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then Chairman of the Examination Committee of the Board saw in this 
an opportunity, and he seized it. 

The result of subsequent negotiations between Dr. Miller's 
office and the Board set in motion a grant application which was funded 
and permitted the study which began in 1964 and ended in 1968 with a 
remarkably altered format for examination. The features of that for- 
mat are recorded in the JAMA of October, 1968. Still, in 1964, '65 
and '66, we were faced with a large number of failures. It has not 
changed that aspect of it. The problem, of course, of the failing candi- 
date for board examination is that it is discovered when it is too late. 

He is now out of training. It is not possible realistically to ask him 
to go back for additional training although we have tried it and now 
abandoned it. It is necessary to discover his shortcomings in the 
course of his training, or they cannot be corrected. , 

About the same time, 1961 and '62, Dr. Vernon Luck, who was 
then a member of the Board, suggested an in-training examination, 
and in 1963 one was implemented. 

The objective of this examination paralleled those of the 
Neurological Surgical in-training examination. 

Our first one was a clumsy effort along side the sophisticated 
instrument that neurosurgery arid the National Boards contrived in 
their first one, but our purpose was the same. We wanted the exami- 
nation to be anonymous. We wanted the reports to go only to the chief 
of the training program, and he to transmit them to the resident, and 
it was not to be available to the Board, and it was not to influence the 
eligibility of the candidate for Board certification, and it remains that 
way to the present time. The in-training examination is currently 
administered by the Academy, a non- certifying organization, and they 
sponsor it purely as an evaluation and teaching instrument in the course 
of the resident's educational period. 

Now, some advantages, however, accrue to the Board. First, 
it is possible for a chief to judge the progress of his trainee from the 
first year through the fourth year . He Can find those areas that are 
weak in individual candidates. He can identify areas that are weak in 
individual programs, and we can find areas generally in orthopaedics 
where the whole field seems to be weak,' and having been identified, 
they can be' corrected. ' .V"' • '?:! '• { • ■ 

It was an excellent area in which to test the quality of multiple 
choice questions, ; for example. It is an opportunity to introduce new 
techniques in examination when they do riot carry the re sporisibilitie s . 



Additionally, candidates learn something about the techniques 
that they will face when their certification examination is taken, and 
I think that each of these hasi been fulfilled very admirably. I think it 
is significant that when our candidates in the 1969 examination were 
debriefed -- and by "debriefed, " I mean after the candidate has taken 
the written portion of the examination, and gone through his two and 
one-half hours of oral examination — each group is, reassembled, and 
a member of the Board interviews them in terms of what they liked or 
did not like about that examination. They indicated that in their opinion 
one of the most valuable instruments that they encountered in the course 
of their training was the in-training examination. 

Now, changing the format in examinations for a board is diffi- 
cult. It requires the alteration of the attitudes that are sometimes 
seemingly set in concrete, and I think George Miller will attest to the 
numerous stormy sessions which his staff and our Board underwent, 
but I think from those stormy sessions there were sparks which illu- 
minated. As a result, not only was a new examination developed, but 
there was a change in the attitude of those responsible for the exami- 
nation. Evidence number one: that attitudes can be changed. 

Now, our Board examines in a peculiar way in that the Board 
itself is not the only cadre of examinants. We give our examinations 
once a year to large numbers, and we need additional examiners, . and 
- we call upon training chiefs around the country to serve in their capa- 
city, and the success or the failure of our new format in large measure 
rested with these gentlemen. If they were sympathetic to it, it might 
work. If they were antagonistic to it, they could scuttle it at price. We 
did not leave this issue to chance, but, rather, assembled all of the 
examiners in a series of meetings in four or five cities throughout the 
country about a month before the new format was initiated. We went 
through its development, we explained the techniques, and we involved 
examiners in actual conduction of examinations in the new format. I 
think the single greatest accolade that I can pay to orthopaedics, and 
I wish to, is ; to these examiners who took it in the spirit in which it 
was offered even though many of them had very serious reservations 
about it, and they agreed to try hard to ’make it work. They did try 
hard, and it did work, and it became its own best salesman at that 
juncture. Evidence number two that attitudes can be changed. 

... ?• Now, the real accomplishments, however, it seems to me, are 
the feedbacks of the training program which have been created. I 
firmly believe that when a critical examination such as the certifying 
examination is given, it will have rather profound influence on the pat- 
tern of the, training of the individual who is going to take it. . You may 
insist; to your residents that the certifying examination really is only 
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an incident in this whole business of postgraduate education, but they 
don't see it that way. To them, it is the most important. Having 
persuaded chiefs that examinations ought to be conducted along the 
lines of solving problems, thij* context of instruction filters back down 
the line, and, now, instead of teaching intuitively, instead of teaching 
simply masses of information, a good many chiefs now teach their 
residents in terms of problem solving. I know of no other way of 
having implemented this most important aspect than the mechanism 
of the examination. 

Well, the upshot of these two changes are these. Now, we 
feel that in order to increase the output of orthopaedic manpower by 
reducing the time required to achieve acceptably competence by some, 
if not all, of orthopaedic residents, we have the instruments to do it. 
We believe that the objective has been defined, it is understandable, 
it is real, it is achievable, and our current in-training and certifying 
examination make the objective measurable. 

DR. JOHN G. FREYMANN: Dr. Gregory, I think we are all 
very excited about what the Orthopaedic Board has done. As you were 
talking, it seemed to me that the next logical step is to extend this to 
quality of care, and I would just wonder if you have any plans along 
this line. It would be very interesting to correlate, if possible, the 
results, past grades, in these examinations with the quality of care 
ultimately delivered. 

DR. GREGORY: I think it is an unanswerable issue at the 
moment, but I would say that one of the thoughts that occurred to 
Dr. Miller's group and ours is really to see what the net effect of all 
of this is on people ten years hence, and that a follow-up study along 
these lines is envisioned. 

I am not certain that the instruments for the measurement at 
this time are as clearly defined as the desire to do so. I suspect that 
measuring the quality of the orthopaedic surgeon ten years later if it 
can be done, will be an indirect measurement of the quality of care 
the individual patient is receiving. r : '?■’ • i- :: 

DR. FREYMANN: But you definitely have this plan. 

DR. GREGORY: This is planned as an integral part of this 
study ten-years hence. ">'■ ••••’.’ > ’>• ; 

DR. EDWIN F. ROSINSKI: You raised, I think, a fascinating 
phenomenon.": People in education and measurement have always told 
formal test conductors that they create examinations and then people 
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teach for the examination. You said that you have now created an 
examination that gets into problem solving, and the result has been 
that the teachers now focus on problem solving. Maybe this sounds 
heretical on my part, but, it is a damn good idea if you can get faculty 
in residency programs to work on problem solving so they can pose 
the problem solving exams. Why not, then, motivate them to do it by 
developing examinations like this that will force them into this kind of 
a teaching situation?. It is interesting. It is a different approach, but 
it may be the only way you can get them off of midcenter. Then, may- 
be, we, can get them to design objectives in terms of this approach. 

DR. GREGORY: I don't know how to comment on that except 
that is the essence of what is happening whether by design, or accident. 
If it was design, I would like to, transfer the credit to Dr. Miller, but 
nonetheless, this phenomenon is evident. I see it particularly and in 
reference to a group in a private hospital that is a part of our affiliated 
system in Dallas. The Baylor Medical Center is a large private hospi- 
tal, and the residents at Baylor are under the direction of physicians 
who haven't really any notion at all about how to go about the business 
of education. It was an osmotic process to a large extent, but it is 
amazing how quickly the chief of this program picked up many of the 
techniques he had learned as an examiner for the American Board of 
Orthopaedic Surgery. As a matter of fa^t, he is so gung ho that he 
has embodied in all of his evaluations the evaluation sheets that Dr. 
Miller devised for us for the business of grading candidates for the 
certifying examination. All of your residents are now evaluated 
semi-annually on the basis of their recall of information, problem 
solving capacity, interpretation of data, etc. Now, this clearly comes 
ringing through to the resident. Many people say to me, "How do we 
teach problem solving? How do you structure a course in it? " 

My answer is, "All that you need for an exercise in problem 
: solving is a resident; with an X-ray in his hand and a question in his 
. mind. " vV.- r >:- ' •••.••••• ,-?r. j- ;7 v' , . , 

DR. CLEMENT R. BROWN, JR. : Do you have any continuing 
education programs in orthopaedic surgery for orthopaedic surgeons -- 
formal or informal? If so, in constructing the curriculum of these 
programs, do you do any surveys of practice of orthopaedic surgery 
in an attempt to gather some data to define needs for your orthopaedic 
surgeons for continuing education programs, and at the same time 
... , feed this back to the boards in terms of content for the board exams ? 

This might be a sensible, thing to do.. We survey practice in 
our hospital, and the kind of things that we find out that the orthopaedic 
surgeons are working in. are the kinds of things that I doubt would be 
asked for in an exam of orthopaedic surgery. 



DR. GREGORY: The issue in point is the in-training examina- 
tion. We said at the outset that it was intended to discover weaknesses 
in candidates and programs. Those weaknesses come shining through 
when the examination results are tabulated because the examinations 
are segmented. There are certain numbers of questions in this cate- 
gory and that category, and you can find out where everybody is doing 
poorly. When we run across this area, we have a disproportionate 
number of questions along these lines the following year. 

. . . . . ; i ■ ■' 

CHAIRMAN HALL: How have you used the in-trair ng exam 
to feed back into your program and to moderate it? 

DR. GREGORY: Although we have rio claim on results so far 
as individual candidates are concerned, the chairmen of our two com- 
mittees — the in-training Examination Committee of our Academy, 
and the Examination Committee of our Board -- are ex-officio mem- 
bers of the opposite committee so that each knows exactly what is 
going on in the other. Each may bring back such information as we 
can use and do use. ■* . • V; \y' ••• •'••!/" 

Yet, on the other hand, we specify clearly in the stipulations 
what things the candidate will be examined in, and we are not free to 
unbalance that examination at the time when it is critical to them on 
certification. We think experimentation along these lines should be 
confined to the in-training examination, which has no responsibilities. 

CHAIRMAN HALL: By- the mechanism of these exams have 
you been able to define the universe better? 

DR V" GREGORY : I doubt we have defined it better . We have 
probably come a little bit closer to the technical aspects of orthopae- 
dics which can be covered by an examination, but we haven't gotten 
any closer to the kinds of problem that Dr. Elling spoke to last night, 
and I don't think you can in the examination. . ' 

DR. MAX MICHAEL: I wonder if you would comment on that 
section.- -’/V-' : : 

DR. GREGORY: It is probably the most difficult of all to 
assess, arid the certificate issued by the Board is riot a certificate 
that attests only to the examination per se. We have other sources of 
information about candidates for certification — his peers in his com - 
munity, his training chief, the chief of the hospital in which he works. 
This material used to be gathered in a rather helter-skelter fashion, 
but it, too; now is, in a sense, programmed. We don't simply ask a 
chief or a peer to write a letter about Candidate X. We ask him to 
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i<- provide a certain specific information about Candidate X, particularly 
1 about his attitudes, how he gets along with people, how he communi- 

cates, how he deals with patients, community responsibility or sense 
of it. This is fed into the profile that will eventually determine whether 
he is a suitable candidate for certification or not. 

Wc try to some extent to discover something about attitudes in 
the oral examination. We have on occa.si.on had examinations with pro- 
vocative questions. The questions weren't designed to disc oyer whether 
or not he knew what a pitcher's elbow in a Little Leaguer was, so much 
as how he would deal with an irate or over -ambitious coach or father 
in a situation like this. Those have been intriguing experiments, but 
they have not always been easy to assess and to grade and to score, 
but this is one venture that has been made in that business of assessing 
attitude in examinations. 

DR. DAVID N. DANFORTH: Dr. Gregory, has your Board 
addressed yourself to the problem of evaluating the quality of the work ■ 
done by the men whom you certified ten or fifteen years ago? 

DR. GREGORY: No, not really. We have spoken of this, but 
1 we haven't gotten around to it. I don't know that we will unless some 

instrument can be developed, and it could be the same instrument that 
would be useful in assessing the current crop of candidates ten years 
hence. 

DR. GEORGE MIXTER: I understand you are giving feedback 
to your Review Committee. Would you enlarge on that ? 

DR. GREGORY: I can't tell you how much feedback goes to 
them other than the results of examinations as we have them at the 
present time We feel at the present time that the feedback is totally 
unsatisfactory, and in a large measure sometimes rather perfunctory, 
that they are rediscovering the same things, that they are remeasuring 
the same things, and that the parameters that need attention are not 
j receiving any attention, and we think part of this is inherent in the 

I system, and it is unbeatable as it exists now. 

; ........ We. are going to make some proposals about changing the system 

' rather radically. ..... ...... 

f . ./; DR. MIXTER: This is..a ; .very large problem,: indeed, because 
the entire structure of accreditation is at stake here. The one prob- 
lem that has not been appropriately explored is the ^riticai mass pf 
graduate medical education in a single institution and the corporate 
responsibility, we will say, of the orthopaedist and the , general surgeon 
and the internist and the pediatrician and the O. B. -Gyn. as a group. 
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Now, every one of these relates tp every one of the others, 
and I wonder in what way this particular aspect could be made a part 
of your kind of endeavor. 

DR. GREGORY: Well, as you know, there is a tremendous 
increase in the amounts of solicitude on the part of various organiza- 
tions for the welfare of residents and training programs right now. 

There are an awful lot of people who suddenly manifest great, deep, 
abiding interest, and there is quite a proliferation of proposed pro- 
grams to correct all of the deficiencies that exist. 

The core cur riculum for surgery is emerging not on one side, 
but two sides simultaneously. What will come of this, I cannot say. 

I don't know how much of this may follow the tenets proposed by the 
Miller Report — to permit the business of regulating the training pro- 
grams to the universities. I can tell you that orthopaedics has the 
unique problem of having more identified affiliated programs in the 
green book than any other specialty that I know of. We have got about 
165 of our programs utilizing university hospitals, children's hospitals 
and private hospitals. Keeping track of all of these is very difficult 
indeed. 

Now, that means if universities take over the business of 
residency education and controlling it in all respects, that non-university 
hospitals are going to have to find a formal articulation with the univer- 
sity or be dropped off the screen. We can't make the sudden transition 
as easily as many others might because we must pay attention to these 
affiliated programs. They are in some respects a significant part of 
the backbone of orthopaedic training programs today. 

DR. MIXTER: Of course, in Canada, they have already accom- 
plished this. By 1972 in Canada, there will be no residency training 
programs which are not affiliated with some university or other. It 
is going to be a long process in this country, as you well know. 

DR. GREGORY: Especially in view of the vying at the present 
time for authority. Is the accrediting body to be a university acting 
autonomously pretty much as they do so far as the MD degree is con- 
cerned, or is the authority going to rest with extramural organizations, 
now principally the AMA Council in conjunction with specialty boards ? 

DR. MIXTER: That is the theoretical problem. The big prob- 
lem in my mind is that our universities are concentrated in a few 
cities. You have New York, you have Chicago, you have multiple 
universities in big cities, but how are you going to divide up the res- 
ponsibility there is very difficult to assess. Furthermore, you have 



an awful lot of darned good training programs in cities that have 
nothing to do with a university. 



DR. HUBBARD: Have you begun to accumulate. Dr. Gregory, 
data that permits correlation between the results of the in-training 
examination and the certifying examination? 

DR. GREGORY: I don't know of any such formal data. Dr. 
Hubbard. We have only stabilized the certifying examination in 1968. 
We don't even have the results of the 1969 examination yet. 

DR. WOODBURY PERKINS: I just wondered. Dr. Gregory, 
in the content of the in-training examinations, is it given exclusively 
by orthopaedists? 

DR. GREGORY: Pretty largely. It takes about a year in its 
preparation. It is put together by a committee which is divided into 
several segments according to the categories of trauma and that sort 
of thing. But these are all orthopaedic surgeons who have some kind 
of an articulation with the training program, either clinical, faculty 
or research. Mr. Lee Natchez, an educational psychologist of the 
American Board of Orthopaedic Surgery, shapes the material the 
orthopaedists bring him into useful, reasonable questions. 

CHAIRMAN HALL: Our next speaker is Dr. Robert King from 
Syracuse, New York. He is Chairman of the Department of Neurologic 
cal Surgery at the Syracuse. 

DR. ROBERT B. KING: Eight years ago for some unknown 
reason a good deal of turmoil broke loose among the neurosurgical 
world in this country with a good deal of concern for our practices 
and policies. The origins of this are not entirely clear. 

Let me fix the base line at that point. The Neurosurgery 
Society had already sent official notice to NIH saying that we wanted 
no federal funds in support of our training program. That was just 
eight years ago, unreal today, but that is a matter of record. The 
Boards at that time, in the view of most, were still largely a closed 
organization. Members were rotating. They were coming from a 
limited source in the neuro-surgical community. , Each of the national 
societies was entirely autonomous. There were, something like eight - 
each with its own membership, rules, inclusions and exclusions. No 
bridge of communication between them, each staking out its own areas 
of .interest. ... . ..... ; v v . ; ... 



The public weal was a term entirely foreign to our conversa- 
tion or considerations except for a very few leaders in the group. 
Education itself, as a process -- a responsibility of the neurosurgical 
community — was largely neglected at the national meetings, and it 
rarely, if ever, appeared as a component of the program. That we 
should respond to change was almost unheard of, for we had most of 
the answers that were essential to our highest performance of a highest 
art. " ’’ ; 

Now, a number of adaptations have occurred in the last eight , 
years. Let me indicate what some of these have been, at least, on 
the national level. 

There are now eighteen training programs supported by NIH 
with training program grants. This represents a major turn. The 
federal tax dollar now supplies primarily the younger and growing pro- 
grams, but not exclusively. The Board of Examiners is now broad- 
based, representative of. all the national organizations and from those 
outside of the neurosurgical world. The Board rotates, and has a 
constant feed-in of new sources. The responsibility of delivering its 
oral examination has been a matter of deep concern to the Board. 

While we are nowhere near as far along as orthopaedics with respect 
to adaptations and examination technique, we, also, use adjunct 
examiners in order that they acquire training. Not so much in the 
present form of the Board, as a resource pool of personnel for the 
Board as the adaptations develop in the next few years. 

The Board is concerned to introduce flexibility into a group of 
programs which are notorious for their rigidity. It is no easy matter. 
These national organizations that I mentioned all agreed unanimously 
four years ago to send, representatives to the Board of Directors of 
one organization, formerly the Harvey Gushing Society, now the 
American Association of Neurological Surgeons, and to name that 
organization as spokesman for neurosurgery in the United States. 

In many ways this means in the world, for half the neuro- 
surgeons in the World are in this country. So all of the national 
Organizations -- in all some fifteen, national neurosurgical organiza- 
tions — now have a way of getting together on matters of common 
interest. Our major interest has been our training dilemmas and 
the evaluation on a broad scale of the world in which we are working. 

An in-training examination was generated under the auspices 
of the then Harvey Cushing Society, and the Board of Neurological 
Surgery. .. x ; 



The purpose of that examination -- although it was generated 
from a concern for the high proportion of failures — was to examine 
the training programs. . The examination was set up to examine the 
training program directors and their programs, just as much as it 
was the trainees. It was also meant to feed the information back to 
the training program directors to jack them up where their training 
programs, obviously, needed additional help. : I hope that we will 
have the wisdom to follow the path which orthopaedics have followed 
in maintaining this as an in-training program examination and not 
allow it to lose sight of its initial intent. 

, The program directors have had a workshop this year, for the 
first time in the history of neurosurgery in the United States. This 
is, again, under the auspices of the joint societies. Another is planned 
for next year. 

At the same time that these national changes are under way, 
local adaptations of training programs are underway on many fronts 
and in many individual programs. 

The requirements of our Boards are thirty months of neuro- 
surgery, six months of basic science -- which is not defined -- and 
twelve months are opened for training in whatever vein the program 
director devises. This allows, then, eighteen months with a good deal 
of flexibility, and thirty months with little flexibility. Some programs 
include three years of basic science ~- which ^ei^s.ibgnc’h. work in a 
laboratory -- plus four years of clinical neurosurgery -- seven years 
in all. They have had a hard time getting candidates down to six 
months of basic science and forty-two months of straight neurosurgery 
and out. The flexibility that is available even within what appears to 
be stringent limitations has depended in some measure on the imagina- 
tion of the program directors. 

In-service testing has been developed on many services. Pro- 
gram directors who are not utilizing this technique. in internal prepara- 
tion are concerned about this. They wonder whether in-service testing 
throws a bias into an actual examination confrontation by developing 
greater;, skill in the examinees in terms of style and rhetoric as opposed 
to information and judgment. I think, frankly, we don't have an evalua- 
tion system set up to consider that problem. 

Evaluation itself generates new knowledge, new experience and 
a change of behavior on the part of the trainee and the: tester, and this 
requires evaluation, and, so, we begin to cycle. : 




It is our concern, furthermore, to move further and to begin 
to evaluate that which we know little about -- not statistics on morbidity, 
not just the currency of practice techniques -- but the responsiveness 
of neurosurgery to the physical, mental, social, economic and spiritual 
well-being of the family units we are called upon to serve: and also, 
to consider our obligations to those who do not call. This is where the 
real difference is. It is here, perhaps, where we measure and evalu- 
ate the least. 

DR. ANGELO P. ANGELIDES: How does one avoid the trap of 
perpetuating sacred cows? A group of individuals says, "This is what 
our boys should be trained for or educated in. " Since they st,t the 
approach arid the requirements, is there any way of avoiding this trap 
and accomplishing what Dr. King mentioned — the relationship of 
training to the needs of the real world. 

Now, it may be the same, but how does one tell? 

DR. GREGORY: The only thing I can see is you need an icono- 
clast every now and then. So you look for one, and you try to wedge 
him in. Once ! he gets there, he is persuasive, he makes his presence 
known. As a matter of fact, the whole thing, is icorioclastic, is it not ? 

DR. ANGELIDES: I am not trying to impugn anyone's methods. 

I am interested in the problem that each of these boards, faces. Maybe, 
Dr. Miller and Ed Rosinski can offer some advice on how we could 
avoid the problem of involvement in special interest fields that may 
represent only a very small part of what they will do in the future. 

This is not to make them tradesmen, but to educate for the real world, 
rather than for soriieone's pet pirojects. Is there any device that can 
be used? I am not being critical. I am asking. 

DR. KING: I wish I had an answer. We try to confront it 
locally. '' >•••• ’ ! V - ' • •: •••• " •. 

CHAIRMAN HALL: Do we have any measurements of what is 
the demand of the real world? I had the opportunity of sitting with the 
President's Commission on Health Manpower. We knew all of the ratio 
findings information, but we really didn't know what those people went 
five times a year to the doctor about, and to design a system without 
knowing the demands is very, very difficult. 

DR. KING: It is more difficult if you don't know what they 
didn't go to the doctor about. ; -i 
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DR. HUGH BUTT: In answer to Angie's question, I think some- 
thing we found very helpful is to include in any testing, evaluating group 
some people who are really out on the firing line. These people bring 
you up short with things that . are real and relative, and" I think you need 
them for a control. Just to take a group of us who are in academic life 
and let us set all of the standards, I think, is wrong. I think you have 
to have the experience of other people. 

! DR. MIXTER: We are dealing really with two problems. One 
is the problem that confronts the Boards universally, and immediately -- 
the evaluation of the people who present themselves for certification. 

Secondarily, we are dealing with the problem of the accredita- 
tion of programs which are supposed to produc. a the people who can 
pass these examinations. 

They are quite different problems, , actually, becaiise the boards 
are dealing with techniques by which they can evaluate the products of 
these programs. The review committees are dealing with the tech- 
niques by which these people can be produced. It is •obvibus that there 
are imperfections in both methods of evaluation. V&y,. % . 

I think there has been on the whole very little feedback between 
the two groups. They function quite separately, and that is unfortunate. 
However, every one of the boards is represented on. every one of the 
review committees, which is as it should be. 

In addition, all of the. review committe.es have representation 
from the Council on Medical Education. The Council, in principle, 
attempts to keep a balance — ard I speak now. to the problem of those 
who are on the firing line — attempts to keep a balance by introducing 
to the review committees persons whose orientations are, perhaps, 
somewhat different from the board representation, .and where a college 
or academy, is concerned, different; from those people,; too. 

. As a matter of fact, if a council representative on a review 
committee is appointed to a board, he is automatically relieved of his 
duties as a council representative. . I don't know how many of you know 
that. So that the council, then, because of its participation in all of 
, ^ the sel activities,-?' has an obligation to be an agency; which is responsive 
to the needs of the people and not the needs of the educational systems 
. and not the s, acred cows and not the establishment.; Whether or not 
the council, by its appointments is able to do this is open to question. 
Hpwe ver, the mechanism is,. there ., ? .... . ; 
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I think the Council should be made aware by those who are 
interested of its obligation in this regard. If there is no feedback to 
the Council, they won't know who they are supposed to appoint, you 
know. So, here again, there is an obligation on the part of the profes- 
sion, generally, to keep the Council on Medical Education apprised of 
the needs of the medical community. 

DR. BROWN: I would like to suggest that the boards are prob- 
ably one step away from the real problem, examining the product of 
the educational programs. And the Residency Review Committees 
are probably two steps away from the real problem in looking at the 
teachers. Until we really get at what happens to patients who are 
managed by all of the products of all of there programs, we won't 
have gotten at what we are really after. 

This is easy to say and difficult to do, but I think this can be 
done, and I think when you begin to do it as it is being done some 
places, you can see how irrelevant some of the training and some of 
the exams are to what happens in the real world in the practice of 
medicine. It makes you very concerned -- it does me -- that there 
are not people on the Board of Orthopaedic Surgery to assist in the 
construction of the exam who are not orthopaedic surgeons. Ortho- 
paedic surgeons do a lot in management of patients -- at least, they 
do in my hospital -- that they could do an awful lot better. And those 
things don't have directly to do with the study of orthopaedics, but have 
to do very much with the management of their patients for whom they 
are entirely responsible while the patient is in the hospital. 

So, I think we should really begin to look at the patient care 
process and feed that back into your Board exams and your training 
program. I think you are one or two steps away from the real problem. 

DR. HUBBARD: There are now two specialty areas that have 
this in-training evaluation. Others are contemplating it. Is it likely 
that by virtue of the introduction of these in-training evaluations, there 
may be reasons for a new iook at the requirements set up by the boards 
of certification? I think some of us have the feeling that these time 
requirements were first established by boards to a certain extent 
because of lack of confidence in the evaluation procedure, lack of con- 
fidence in the examinations. Now, there is reason forbore-- confidence •• 
in the evaluation and the examinations, and if an individual shows up 
as superior by virtue of his evaluations during his training period, is 
there still the same justification for holding him to a predetermined 
period of -- I think you said seven years. Dr. King -- or whatever the 
time interval may be ? 



DR. KING: Our boards are now prepared to consider a recom- 
mendation by a program director that a man may have accomplished 
his basic science and his basic experience in the neurosciences even 
during medical school and can be relieved of an obligation to repeat 
that period of time in his post-doctoral period. The basic science 
requirement may be fulfilled during medical school because of changes 
in the school curriculum. This also becomes another argument for 
turning major responsibility for these programs to universities. They 
can pull the time base out from underneath the ppst-doctoral period. 

CHAIRMAN HALL. I might also say parenthetically this forces 
the program director to be self-penalizing because you are getting rid 
of the man you would like to have around most, who provides the leader- 
ship and the things that make life much more pleasant. 

DR. GREGORY: Well, the next step in the American Board of 
Orthopaedic Surgery is precisely in this spot, but not left only to chance. 
There will be careful assessment of the nature of the program that pro- 
duces people who may be eligible for certification at a shorter time 
than the conventional period. It is this element that we would like to 
extract and reintroduce into other programs if it is possible to do it. 

CHAIRMAN HALL: Our next speaker is Dr. Hugh Butt, and 
Hugh has led in many things. One of the things he has led is a self- 
re-examination program that was referred to yesterday. I think that 
we are indebted to Dr. Butt's enthusiasm in the development of this 
important tool. 

DR. BUTT: The College of Physicians has been at this education 
game for many years, and, in fact, in the Charter of 1916, continuing 
education is mentioned as its prime purpose. 

Now, it has done the usual things that any large society --about 
fourteen, fifteen thousand members • — has done. It has its annual, 
meetings, and it has its regional meetings, and it has postgraduate 
courses -- which it was one of the first to start in this country — and 
this seemed all right for a while, but its Educational Committee started 
looking into this and talking to members, and members were very 
frustrated because they felt a lot of biomedical knowledge was passing 
them by, that they were not applying this new knowledge to practice. 

In talking to literally hundreds of these people throughout the councry, 
we found that this is essentially what they wanted, something to close 
this gap, if possible. 

Well, the Committee looked around to; see what was going on in 
continuing education, and we found a primordial ooze. This was exactly 
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where we put our feet, and we are not out’ of this ooze, but we are 
trying to make a little progress. 

We found that the things that all of you know were going on — 
the television, the tape, the sound tapes. Postgraduate courses by 
universities rarely begin at the convenience of a practicing physician. 
Often when he got there, there were poor speakers, poor slides, poor- 
content, a great waste of time. In talking to these men, though, the 
thing that they really didn't know, was just what they didn't know. This 
gave us the idea that, maybe, some sort of "self-evaluation" was 
needed. We began with some major premises. 

The first one -- physicians want to learn and increase in 
excellence. They would like to know their deficiencies -- providing 
no one else knew them. All of us have some paranoid trends and need 
careful reassurance. 

This last sentence usually brings forth a lot of smiles, but in 
talking to several nationally respected psychologists and psychiatrists, 
we found this is true of all people -- whether it be state bar exams, 
national boards or whatnot, in which other bodies have to certify them 
and which may affect their income and practice. We were advised that 
because this was the first time, on a national level, that doctors were 
being offered an opportunity to assess themselves, that we not try to 
gather information. As a matter of fact; the psychiatrists felt that the 
exam or the self-evaluation, as we prepared it, would cause quite a 
bit of depression anyway. 

Our idea was rather simple -- the member would volunteer to 
be appraised. It would be taken at his convenience -- with or without 
help. The answers would be sent to a bonded firm. That means that 
Hugh Butt wouldn't know, or the College of Physicians wouldn't know, 
or NIH wouldn't know. The bonded firm is a group of certified public 
accountants, and the name and scores of the testees were erased after 
they had been graded by computer, and the results were returned by 
confidential mail. 

When this is discussed someone always says, 'Well, gosh, it 
is kind of a wa •:*?’ of money and effort to go to all of this trouble. You 
are not getting any !i!ata for the college or for the educators. " This is 
true but we were advised very strongly not to. 

The real purpose of this first baby step was to get the confidence 
of the members of our organization and other physicians in the country 
and assure them that the College was since roly trying to help them. 
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We arbitrarily divided internal medicine into about nine areas, 
and you can guess what they were. We selected outstanding committee 
chairmen of each of these areas, elected five people, one of whom had 
to be someone in general practice of internal medicine, and the chair- 
men were told to include in these questions scientific knowledge that 
has developed in the past decade that could be directly applied to the 
patient. This wasn't to be a recall of information and data bank, but 
simply to how could you apply this material that had been collected. 

It was available, and did you know that it could be applied to a given 
patient situation, and they developed about seven hundred questions 
with illustrations, some in color. A nominal cost of ten dollars was 
made to the members. We estimated of our thirteen, fourteen thou- 
sand members at that time, that there might be five or six thousand 
who we would consider; were eligible. This excluded pediatricians, 
psychiatrists, scientists in the club and also, maybe, men past sixty- 
five, although to my surprise many, men past sixty-five took the, test. 

Well, much to our surprise in the first two weeks four thousand 
people signed up, and thirty-five hundred people returned it on time. 

Now, since then, this has been made available to non-members, 
and over ten thousand of them have been purchased, and they are being 
used in many ways — as a teaching aid in both medical schools and 
universities hospitals. 

I can only say that we are quite convinced that this is one, 
simple, important method of continuing education. It is by no means 
the whole answer. It is just one tiny p: rt, as we all realize. 

I am quite convinced, as others, that physicians want to learn, 
they will work to learn if you can do it with dignity and with reassur- 
ance that it will be private ana voluntary. 

DR. THOMAS H. BREM: Hugh, I talked to quite a number of 
.people who have taken this examination, and every one of them has 
said that he has felt rather ^frustrated and unhappy because he really 
didn't know where he stood in the scheme of things after he got his 
results back. Now, most of them, in fact all of them, that have talked 
to me about the examinatipn have said they wished they had' sent a curve 
back so they would know approximately how well they did relative to 
the other thirty-four hundred. ' vJ' \ '.,.y , ;w ./?! ; .. . . , 

DR. BUTT: Let me comment on that, Tom. ; We were advised 
that this would be a complete failure if we sent out a prospectus and 
asked the people to fill out a lot of data about themselves, whether you 
have had your Boards or not and so forth. It would be the kiss of death 
just as much as^Lf we had written down, "This is supported by NIH. " 
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Nov, we have received hundreds and hundreds of letters, 
mostly complimentary. This is not mentioned very often, but in going 
around the country and talking with people, I find, too, that deep inside, 
this is what they would like to know. So, now we are meeting to start 
to prepare another one to begin in 1970 or '71. I think now we can 
make this step and get this data and get curves and get information 
back. 



One of the things that also deterred us at first -- the press 
was very inter ested in this when it was first announced to them. They 
thought it was wonderful that a group of physicians were going to try 
to evaluate themselves, but as soon a ii they found there would be no 
grades available they lost interest. In fact, only one newspaper reported 
this as something that was good that the doctors were doing. Since then 
I have talked to some good newspapermen, and they tell me that no news- 
papermen would be interested in this unless they can get bad news out 
of it. I. think they are right. George Miller has, I uiink, some good 
ideas about how we might present this as a research project and just 
publish it in the literature without announcing it. It is not anything 
new now, so I don't think the press is going to be interested anyway. 

Just publish it in the Journal of Medical Education or the Annals , and 
let the curve be, and let them interpret them any way they want. 

DR. AIMS C. McGUINNESS: The curves wouldn't be worth 
very much anyway if we didn't know who got what kind of help and so 
forth. 

DR. BUTT: This is another thing Dr. Hubbard has pointed out. 

If you really want true hard core data, it has to be monitored with time 
and content. The only other way I know we can do it is give part of it 
at the Annual Meeting. You could give it to people in an audience. 

Give them the exam for two hours and monitor it and feed it into a 
computer. 

DR. HUBBARD: You might be interested to know we have had 
discussions with the American Psychiatric Association on very similar 
lines, and they have gone down precisely the same path. They have 
not been willing to face up to the idea of a scored examination at the 
first step. The pediatricians are also doing it. It is the "in" thing. 

I don't think I am apprehensive of the press. I think this depends 
entirely upon how it is presented. ' ' 

We distributed a frequency distribution for the neurological 
surgery grotip. An individual Icnows where he stands on the curve, but 
this is something very different from letting the press know that "x" 
per cent of the physicians of the country don't know a certain amount. 
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The press can’t do very much with where an individual may stand in 
a frequency distribution. 

DR. ROSINSKI: I think it is a question of what kind of data and 
how~it is going to be used. Aims’ point was a good one. He was saying 
it was a natural instinct of an individual to want to know how he did in 
relation to somebody else. You are still motivated to take examinations, 
and they are competitive in spite of what you do in them. It is just 
natural in you. You know how well you did on the exam yourself, but 
it is natural to wonder how other people did. These kinds of data could 
be provided. 

DR. McGUINNESS: How well should he have done? 

DR. ROSINSKI: That is carrying it one step further. 

DR. BUTT: I might say one of the important things in this 
self-assessment, we tried to get away from the word "exam. " We 
send out references with each question, and many of the men have 
found this very useful. Over a period of time, six months or a year, 
they could look up things that they missed. We have had suggestions 
that this is really not the answer either, that we should send out 
abstracts of these references, and I think this is quite important. It 
is very difficult for many physicians in small communities to get to a 
big enough medical library to find these references, and if we are 
really going to help them learn, help them close their information and 
knowledge gap, then, I think we ought to make it something easy for 
them. 

DR. GREGORY: Aren't we skirting a little bit the question of 
a passing grade, which brings to mind at once the specter of recertifi- 
cation? How much of this really has its motivation in producing some 
implements that extraneous bodies may find acceptable for recertifica- 
tion? Are you thinking along those lines, really? 

DR. BUTT: Well, we have been accused of opening up a 
Pandora's box in this line, but as you well know, recertification has 
been talked about for twenty years. I suspect it is going to come in 
some form, and I would hope if it does, that it is controlled by people 
like there are in this room, rather than men from some government 
organization where there wouldn’t be any flexibility. 

This human feeling of wanting to know how you stand is a natural 
one. I don't think you can do away with this. I think it is good. 

DR. ROSINSKI: I agree. I think it is. 
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DR. NORMAN STEARNS: I think it is interesting to note that 
the Academy of General Practice came to the Postgraduate Medical 
Institute in Boston a couple of years ago and asked if we would produce 
for them a self-assessment examination similar to the lines of Carl's 
physician examination. The motivation for this examination may have 
been varied. Initially, some people thought they just wanted to prepare 
for an examination in family practice, but it was made eminently clear 
to them that this was not the thing to do, and it was not really possible 
to do, that a self-assessment examination might even be bad. 

Nevertheless, two academies, Ohio and Connecticut, said they 
had a higher purpose in mind. They wanted to provide an educational 
tool. The idea wasn't to find out where you stood, really, but to help 
to teach by means of the examination. 

So, we devised an examination for them in which the answer 
to each .question provided a piece of information, and in the answers 
we provided not just a reference, but a notation, so that there was a 
reason why the answer was correct. 

The information from this examination has not been discarded. 
The Academy does not intend to give out individual information, but 
one objective, certainly, is to identify those areas of need for more 
education. In the areas of weakness, programs will be developed to 
guide in the fulfillment of a higher educational goal in the specific 
areas where demonstrated need has been evidenced. I think it is 
interesting that there may be different kinds of approaches to the con- 
cept of the examination itself. It is also interesting that they couldn't 
get this examination done for them in other places. 

I might have an opportunity speaking to your question about the 
library, of big reference sources, there will be an article coming out 
in the February 27th New England Journal this month on a core library 
for practitioners which we have compiled. Most necessary information 
will be in this core library which is absolutely feasible for every com- 
munity hospital, the journals and texts costing less than two thousand 
dollars. 

DR. BUTT: I would like to ask a question of this body. I don't 
know whether George Miller or John Hubbard or both might Comment. 

I would like to know if this is the first step in the swamp, where do we 
go from here in really evaluating? Here we have a large number of 
people out seeing patients and then, obviously, want to be evaluated 
and know where their deficiencies are. 

What is the next step in evaluating this so we can help them 
more? What do we do? Let's get it off the blackboard. 
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DR. HUBBARD: I think we need to keep rather clearly in mind 
the difference between the objective of continuing medical education 
and the objective of assessment. The College of Physicians and these 
other groups have expressed the desire to get real data from their 
membership as to where their weakness is to guide their postgraduate 
training programs. If you want this kind of information, I think it has 
to be an examination and an examination under supervised conditions 
that will give you reliable data. This, I think, is something very dif- 
ferent from the kind of self-assessment activity that is really aimed 
at the objective of education. 

DR. GEORGE E. MILLER: What this provides is to help some- 
one identify a need to know. Until he has this, it is unlikely that he is 
going to embark upon a learning program, but learning is individual. 

It seems to me unlikely that a national organization is going to be able 
to provide an educational program for thirty-five hundred individuals. 

It may provide resources. It may provide references and material, 
but the learning ultimately is going to take place at the individual level. 
It is for this reason, it would seem to me, that this organization — the 
Association for Hospital Medical Education --is one that must direct 
its attention to this question at the local level where the opportunity to 
work with individuals is greater than through a national organization, 
but utilizing data that may be provided by a national organization that 
has an opportunity for a far larger sample than would be possible at 
the local level. 

At the same time, however, the problems in each institution 
or with each practitioner are to some extent unique, and, so, the sort 
of thing that several members of this Association -- I note particularly 
Clem Brown and Bob Evans trying to gather local data about physician 
performance as a self-assessment method -- should be incorporated 
with the broader accumulation of information, the identification of need, 
the specification of objectives for an educational program and appoint- 
ment of reference against which they can make some assessment of 
the extent to which those objectives are achieved. 

DR. McGUINNESS: Each physician probably has a relatively 
small need of information for the majority of his cases . . . 

DR. BUTT: I don’t agree. - 

• DR. McGUINNESS: Some of it you store in fragmentary manner 

in your computer. You don't need to be necessarily able to recall that 
indefinitely. r You need to remember that the information is there and 
where you can go if you have to go and look it up, • ; 



DR. BUTT: I suspect you are not being fair to yourself. I 
think your computer is using all of these things every day. Every 
patient you see you have to in some way eliminate numerous things. 

DR. McGUINNESS: There are certain things, but there are a 
great many details on one of these examinations that you do not have 
to carry around with you. 

DR. BUTT: Well, I see what you mean. 

DR. McGUINNESS: That you would have to carry around if 
you were going to get a reasonably good score on that examination. 

DR. BUTT: This assessment was not for recall of dosages of 
drugs and things of that sort. This had to do with the application of 
modern knowledge to the practice of internal medicine. After all, we 
all have data banks and computers to keep all of those little things. 

Ours is just not big enough. 

DR. FRANK L. HUSTED: It would seem to me that Dr. Butt's 
objectives of a particular effort may well modify or determine or pre- 
determine the way in which data are handled, and the way reports are 
made, and the way specific examinations and devices are developed. 

I would point out specifically the objectives in the total process, 
one of which was immediate: you wanted to gain the confidence of your 
colleagues so they would respond with some degree of veracity and in 
sufficient numbers as to make your study valid. This is an impeding 
objective. When this has been achieved, you can reduce this com- 
ponent, and thereby introduce other kinds of issues into your examina- 
tion process which heretofore were not possible, and, therefore, taking 
the next baby step. I would caution against taking a giant step at that 
point simply because you have this gain. Don't be overconfident by the 
gain scored, the gain realized, but take another baby step by adding 
another dimension. 

DR. BUTT: Very good, I agree. • 

DR. WINSTON R. MILLER: We are seeing the emergence now 
in this Association of Directors of Medical Education throughout the 
country of an expanding interest in directing continuing education at 
the community level, and we ought to exploit this possibility and have 
some plans for doing sol To what extent could the next step of the 
American College of Physicians' self-assessment program provide a 
very much needed nucleus or multiple nuclei of topics that could then 
be distributed to Directors cf Continuing Education in community 
hospitals as a definition of need for postgraduate education? 
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One of the principal problems that always comes with every 
DME when he talks to his medical staff is what do you need to know, 
and there is a lot of argument about this. So, any facts we have to 
present greatly facilitates local efforts. 

DR. BUTT: The AMA, you remember, started out looking 
into continuing education, and they came to some conclusion that a 
group of doctors could sit around in a community hospital with a medi- 
cal educator or director and decide what they needed to know. We 
don't think this is possible, and I think you have got to in some way 
evaluate yourself privately to find out, because you frequently are 
humiliated to find that you don't know near as much as you think you 
know, and it is a very embarrassing, but very stimulating thing to 
find out privately. 

CHAIRMAN HALL: I have bought the examination with the 
cooperation of Dr. Butt and Dr. Rosinski, and I gave it to my gradu- 
ate students in medicine at our institution, and I found some common- 
alities in which my men were not performing in the nine areas that 
you had separated them out. This gave me the ammunition I needed 
to go back to the faculty and say, "We have to beef up in this area. " 

Our next speaker is Dr. Thomas Brem, who is relating his 
concept of the Advisory Board of Medical Specialties in its relation- 
ship to evaluation. 

DR. BREM: Last night as well as today, I think that the 
comments made it quite apparent there are several different levels 
of evaluation. The boards, themselves, are concerned themselves 
with evaluation of individuals, the Review Committees are evaluating, 
really at another level, and there are other programs, resources and 
so forth at various institutions attempting to put on a good educational 
program. 

My report is involved in another level of evaluation. I don't 
mean to say one is more important than the other, or one is higher 
than the other, but they differ. The level at which the Advisory Board 
operates is really the national level involving what we think is the total 
need of the country, the consumers of medical care as well as the pro- 
viders of It, and, as a consequence, this sort of body should be rather 
broadly constituted; Here I might mention Dr. Elling's suggestion of 
having people from outside of medicine -- knowledgeable, well-informed, 
objective groups -- who can survey the whole territory and give us, 
perhaps, some little ideas as to what are the directions we should be 
going. V v>:;. h- ■ " V . 
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The Advisory Board consists of representatives of all of the 
specialty boards, in addition to other organizations that are concerned 
with and involved in graduate medical education. The American Hospi- 
tal Association is represented on the Advisory Board. The National 
Board for Medical Examiners has its representatives. The American 
Federation of State Boards. A couple of others. The Educational 
Committee for Foreign Medical Graduates participates, and the Asso- 
ciation of American Medical Colleges. There are six, I think, 
altogether that are represented on the Advisory Board for Medical 
Specialties, and I think this is an indication of the recognition of the 
Advisory Board that it does have quite broad responsibilities that 
involve a good deal more than simply evaluating individuals or evalu- 
ating programs. 

The major evaluating function of the Advisory Board is the 
receiving of proposals for the establishment of new specialty boards, 
and this has been a primary function for many years now. 

The process is something like this. A group that feels that it 
is desirable to establish a new certifying board with examinations, pro- 
grams and so forth, submits its proposal to the Liaison Committee. 

It is a committee consisting of representatives of the Advisory Board 
and the AMA Advisory Board on Medical Application which reviews 
preliminary applications and attempts to make a decision as to whether 
the move is desirable for the community as a whole, for medicine, for 
medical education and for the improvement of standards of practice in 
the country. 

Well, I will submit that this is by far the most difficult sort of 
evaluation of all. The Resident Review Committees are next, and I 
have spent a number of hours on the Resident Review Committee on 
Internal Medicine. I note it is exceedingly difficult to evaluate those 
programs, but it is duck soup compared to evaluating the greater 
problem as to whether a new specialty board should be in existence, 
and whether this is really in the public interest. We make decisions. 
Certainly, oftentimes, they are not received very happily by the pro- 
posers but they are made in vast conscience, and with a great deal of 
study. • ' 

Now, right at the moment, as an example, there are three new 
boards being proposed before the Liaison Committee; One of them 
has its application for a new Board in Family Practice. 

We have an application from the Nuclear Medicine people who 
wish establishment of a conjoint board composed of representatives of 
the pathologists, radiologists and internists. This is a preliminary 
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application that will be reviewed, and it will be up to the Advisory 
Board and the Liaison Committee and the Council on Medical Educa- 
tion to make an evaluation and decide whether this is desirable from 
the national standpoint. 

Third, an application submitted by the American College of 
Clinical Immunology and Allergy which is additional to the present 
sub -specialty board of the Board of Internal Medicine in Allergy. 

It has been debated at considerable length as to whether it is 
appropriate to set up a second board in allergy. 

There are many changes going on, in graduate medical educa- 
tion. I find that this past year — since I have been an officer of this 
Advisory Board -- I have been involved in a dozen meetings to discuss 
the Commission on Higher Medical Education and Graduate Medical 
Education. Some of the functions of the Advisory Board will be trans- 
ferred, I expect, and that, presumably, would be the organization that 
would take the broad look at the needs of the country and how the vari- 
ous organizations involved in our specialty or graduate programs are 
functioning. 

There will be changes, I think, in the Advisory Board, which 
has been a nebulous s art of organization, without much in the way of 
authority, and I think that the members of the Advisory Board and the 
specialty boards that are the parent organizations are beginning to 
feel, too, that there has to be an organization that ties them together 
more closely and makes some decisions which will be binding on all 
of them. 

You still have a good deal of parochialism among our special- 
ties, but I sense, as Dr. King mentioned about the neurosurgeons, that 
all of the specialties are beginning to become a little more flexible and 
reasonable. 

DR. ANGELIDES: Has any effort been made to try to coordinate 
overlapping requirements or functions of specialty boards so that it 
leads to shorter training periods, reduction of faculty time, avoidance 
of repetition within the same hospital of the same course material? 

You alluded in the latter part of your statement to a group which would 
act as a clearing house for coordinating all these evaluative criteria 
and would act as a focus by which you could splash over to other boards. 
This is a very dynamic role. I don't know: if you are prepared to under- 
take this, but has there been, at least, any thought given ? 



DR. BREM: There has been thought, Angie. Since I have been 
on the Board, and long before that, the Advisory Board has not really 
paid a great deal of attention to the content or the program of an indi- 
vidual Board once it has been approved. 

I think we all think this is one of the weaknesses of the Advisory 
Board, that we should be concerned with the content of the programs of 
the various specialties. I don't think any of us believe that we should 
be designing curriculum, but we should be taking an overview of the 
practices and policies of the constituent boards. We hope that that will 
be one of the authorities that will be delegated to the Advisory Board 
in this rather changing situation. 

To me it is rather disturbing that once a board is approved it 
can do almost anything it wants to do without having to come back for 
reapproval of the Advisory Board as a whole. It strikes me that this 
is contrary to our concept of corporate planning of educational pro- 
grams. At least major changes in policies, in requirements and things 
of that sort proposed by any individual board should come back to the 
Advisory Board as if it were a new program. 

This may come about. It depends on how much authority indi- 
vidual boards are willing to delegate to the overall Advisory Board. 

DR. HUSTED: You set no criteria or limitations once approval 
is made. 

DR. BREM: Pretty much so. 

DR. HUSTED: No pattern they must follow or recommended 
patterns. 

DR. BREM: Pretty much on their own after that. 

DR. KING: Dr. Husted pointed out in the context of his argu- 
ment, that among other things external agents are not adequate units 
for determining educational programs aside from the school as a unit 
of scholars. That is where he put the "buck. " 4 ; y ' 

Nowj obviously, Mr. Millis has read this, arid riiany of us have 
deep concern that this, in fact, may be 'true. How do you weigh this 
philosophy with your staterrient that, perhaps, an Advisory Council 
overriding the Board -- which is yet one more unit detached further 
from the universities and the educatibrial uriits -- should have power 
to control and direct and design and modify educational programs. 

That is a sticky wicket. I would like to hear it discussed. 
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DR. BUTT: May I ask my question before Tom's answer? I 
think he can answer them both. Jack. 

CHAIRMAN HALL: All right. 

DR. BUTT: If you can do as Dr. Brem said -- have some 
control -- then, the status quo does not remain the same sacred cow. 
They are out after two or three years on the Commission, out of the 
boards, and in this way, you can answer part of your question, a 
dynamic infusion all of the time of academic people, and you won't 
have what we have today in some boards -- people who stay fifteen 
and twenty years. 

CHAIRMAN HALL: Tom, would you like to address yourself 
to Dr. King's question? 

DR. BREM: I don't know if I can adequately answer the ques- 
tion. Yes, this is one that is going around and comes up at most of 
these meetings, and that is the question of the role of the university 
in graduate medical education. A lot is said about this, and at the 
AMC Workshop on Graduate Medical Education last fall in Washington, 
D. C. , as well as in a meeting in Houston later, a rather strong con- 
sensus was voiced that a great deal of what is being done now by boards 
and review committees, particularly in regard to the design of pro- 
grams, is being left up to the universities. 

Well, I don't know whether there would be great objection to 
this, but I do believe that at the moment the universities are not ready 
to do this. They might be in the future. 

An example, for instance, was the interest of the universities 
that was displayed at the Council on Academic Societies Meeting. I 
think a half-dozen university presidents were invited, and John Millis 
was the only one that turned up, and, yet, this whole thing was on the 
role of a university in graduate medical education. I don't really think 
the universities, themselves, have much interest in it. The medical 
schools may. But I find the medical school itself is not sufficient. 

This is a university function, an obligation. So, even though this 
might be desirable, I don't know whether it is realistic at the moment. 

I do think it is tremendously important for the Advisory Board 
on High Commission on Graduate Medical Education to encourage flexi- 
bility and experimentation in the design Of programs and not be too 
terribly worried about the format of it, about the curriculum. More 
important are the resources. Does the institution have the basic 
resources to put on a modern educational program in surgery or 
orthopaedics, O.B. , medicine or whatnot ? . 
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DR. GEORGE MILLER: Tom, you have indicated that the 
deliberations of the Advisory Board are primarily addressed to educa- 
tional issues, questions and decisions. I would raise the question of 
whether this is, in fact, accurate, or whether their decisions and 
actions represent a kind of vector resulting from a variety of political 
forces without much regard for educational principle. 

I ask this question for the simple reason that if education — 
which means facilitation of learning --is the principal objective of 
graduate programs, then, how can the Advisory Board justify the 
approval of any residency or specialty requirements whose base is 
upon a temporal requirement for training, because if learning is the 
goal, then, the time it takes is the time it takes. 

I wonder the extent to which the Advisory Board might not 
profit from addressing the question of justification for any kind of 
temporal requirement for residency training. 

DR. BREM: I think you are perfectly right, George, and this 
is another one of the authorities, really, the Advisory Board has never 
had. The question has been asked as to why there are absolute time 
requirements. We haven't been in a position to say that has got to be 
a flexible thing, and, really, you are talking about individuals, and they 
mature or learn at different rates, 

DR. GEORGE MILLER: If I understand it, the Advisory Board 
does approve new boards. 

DR. BREM: It does that, yes. 

DR. GEORGE MILLER: And the function of the Board is to 
establish the competence of those specialties, and that competence is 
based upon time of training, and, thus, the Advisory Board is, in fact, 
using this as the justification for the approval of a new specialty. 

DR. BREM: Part of it. Yes, it does. Well, it has, of course, 
accepted the rather inflexible time arrangement as part of the whole 
thing, and it may be at fault in this. You are perfectly right. 

DR. MIXTER: There is one aspect that has not been mentioned, 
and that is that there is a certain peculiarity of American medicine — 
that no aspect of medicine in this Country is a federal responsibility. 

In every other country in the world, the entire course of medical edu- 
cation, including licensure and specialty certification, is a central 
governmental function.' That is not true in this country. There had 
been no definitive article in the literature devoting itself to this aspect 
of the problems that we confront: - s-\ 
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There has been, however, in the Cornell Quarterly Law Review 
a broadly based article on the subject of accreditation of educational 
programs. It does apply to graduate medical education although gradu- 
ate medical education is not mentioned specifically in the article. It 
is a very interesting concept that in American law any organization 
which sets itself up as an accrediting agency after a certain length of 
time is accepted by the public and generally becomes de facto an 
accrediting agency. It is a rather strange and peculiar setup, and 
I think anybody concerned with these problems ought to familiarize 
himself with the obligations which devolve upon these self-constituted 
accrediting agencies. It is a terrifying affair, and our law department 
is extremely concerned about it, because these agencies, although they 
have never been challenged, are open to suit, and one such suit may 
bring down the whole structure down around our ears. It is astonish- 
ing on what a tenuous basis this whole accrediting structure is founded. 
It really is terrifying,; 



DR. RAY ELLING: It is quite fascinating to note -- and I: don't 
think it is simply the role of government that is the difference -- that 
the development of medical specialties in Britain has been very differ- 
ent than in this country. The whole division of labor, in the field of 
medicine is quite different; the functions performed by different groups. 
There might be some place here for some analysis, comparisons as 
to how this has occurred historically in these two different settings. 



DR. MIXTER : Canadians, too, have an entirely different 
structure. Most interesting. . . 



If Jack Nunemaker were here, he would certainly comment 
about his recent experience in a medical school survey in Canada to 
which the Council on Medical Education of the AM A does contribute as 
well as the Royal College. .. 



At that particular survey, not only was the school accredited, 
but the graduate medical programs. The residencies, also, were 
looked at, and a report sent back to the Royal College — which is 
their accrediting agency — and this whole package was looked at at 
one time. Most interesting. I think, maybe, our Canadian neighbors 
have got a better system than we have in many ways. 






DR., STEARNS: Dr. Mixter, I am just not sure what you are 
saying are the recognized accrediting organizations in this country. 
Are there ones other than what we are talking about tndav? 



DR . . MIXTER In the allied medical field, yes. ; x t-y, 
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DR. STEARNS: Not in medicine. 



DR. MIXTER: In medicine we have set up our own accrediting 
agencies without anybody except the people involved saying that is what 
they want. 



DR. STEARNS: You are not inferring that a governmental 
system would be better ? 



DR. MIXTER: No, I am not at all. I am just saying that it is 
different from anything anywhere else in the world. 



DR. STEARNS: By the will of the people, you mean the doctor 
people or the people people ? • 



DR. MIXTER: The people people are the ones who could object. 
The doctor people are the ones who are least likely to raise some 
objection. People people don't know much about this and, as a matter 
of fact, doctor people don't know much about it either. 



DR. BROWN: I was wondering if Dr. Brerri could give us an 
example of the kinds of data that the Advisory Board collects to facili- 
tate their decision-making process with respect to the establishment 
of a new specialty, particularly, with respect to consumer needs. I 
think you stated directly that this is one of the concerns of the Advisory 
Board, the concern with the needs of the public, consumers, society. 
Would you give me an example of the kinds of data you collect to facili- 
tate your decision-making process in this area? 



DR. BREM: I really don't know that we have any organized 
data on the demands, desires, of consumer bodies on which judgment 
is made. Opinions, but not data. I am not even sure/ really, that 
there are data of that sort. 



DR. BROWN: Some people think there are, I guess. 



DR. BREM: I am not sure I know how much it would be worth 
to have a labor representative, for instance, give his opinion relative 
to a health program, or as to whether it would be desirable to have a 
Board of Nuclear Medicine. Really, I don't know where you would 
derive meaningful and helpful data on those things. But we don't get 
it,';'?to answer directly your question. Dr. Brown; ' ;> v , ; 



DR. HUBBARD: I spoke earlier of the question of time require- 
ments. George Miller has; T think,- gone one step further and suggested 
that the' Advisory Board should withdraw — or the specialty board should 
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withdraw all time requirements. I don't think I would go that far. The 
reason that I would not go that far is because, as we were saying last 
night, there are certain things that are very difficult, some things that 
can't be measured in the competence of a physician, and I think there 
is something known as maturity that comes with experience that is 
important to include in the requirements for certification. 

I would like to know that the surgeon that is operating on me has 
had certain years of experience doing what he is going to do to me. So, 
I wouldn't go so far as to propose to the specialty boards that they with- 
draw all time requirements. I do think they ought to be more flexible. 







DR. McGUINNESS: Some are planning to be, and, some, 
actually, are beginning to be. 

CHAIRMAN HALL: The next presenter will be Dr. Robert 
Moore, Secretary of the American Board of Surgery. 



DR. ROBERT M. MOORE: The American Board of Surgery, 
Inc. , was chartered in 1937- It was the fifth of the surgical boards 
to be organized. 

, It was; chartered under the laws of Pennsylvania and organized 
under State laws. 



At the beginning the Board had thirteen members, all appointed 
for six-year terms, no man being allowed to serve a second term 
except the full-time Secretary-Treasurer whose membership on the 
Board is determined by the leiigth of his office. The American Surgi- 
cal Association, the American College of Surgeons, and the Section 
on General Surgery of the A. M. A. , each nominate four of the current 
eighteen members of the Board. The New England Surgical, Southern 
Surgical, Western Surgical, Pacific Coast Surgical, Central Surgical, 
and The Society of University Surgeons have one representative each, 
serving a six-year term. The Secretary-Treasurer is the nineteenth 
member and three years ago we brought to our office a full-time 
Associate Secretary-Treasurer who has since been made the twentieth 
member of the Board. ■ ; r.;; ’ ••■■■■■■■■ 



After the Member completes his six-year term of Active Mem- 
bership he becomes a Senior Member and usually helps with our Part 
II (oral) examinations for a number of years. A great deal of interest 
andToyalty is attached to this service he renders us. 

v ^ . r It is obvious that we are going to have to increase the Active 
Membership because of the increasing demands upon our Members. 
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Prior to 1950 there was a great deal of confusion as a result 
of two lists of approved surgical residencies being published annually, 
one by the American College of Surgeons and one by the Council on 
Medical Education (and Hospitals) of the A. M.A. These two lists did 
not agree; one year a Training Program would be put "on probation" 
or removed from list while at the same time it was being added as a 
newly approved Program on the other list. Finally, in the late i.940's, 
confusion became so great that the American Board of Surgery pre- 
vailed upon the other two organizations, the College and the Council, 
and induced them to join with it in forming a tri-partite residency- 
review committee, and at that time each of these three organizations 
relinquished whatever rights it had previously held in approval of 
surgical training programs, assigning these rights to this new tri- 
partite Committee -- the Conference Committee on Graduate Training 
in Surgery (which recently changed the term "Graduate Training" in 
its name to "Graduate Education"). 

This Committee at the moment has under its charge 492 approved 
Training Programs in Surgery in the United States and its territories, 
which provide posts for 6, 589 surgical residents. Ordinarily in each 
of these Programs the Chief of Surgery can send one of his junior 
residents to any other hospital he chooses on a rotation up to six 
months without reference to the Conference Committee. In the senior 
residency year, however, a rotation of even one month away from the 
approved parent institution must have the Conference Committee's 
prior approval. At any level of training such a rotation for more than 
six months also must be approved by the Committee. 

As to the requirements for certification, when our Board was 
established there was a five-year requirement, three years of formal 
training followed by two years of practice or additional training. After 
ten years, in 1947, the training was divided into two Programs: the 
Group I Program in which the candidate trains in a Type I (four-year) 
surgical residency for the entire four years,, and a Group II Program 
in which the candidate trains in a Type II (three-year) surgical resi- 
dency for three years, and then takes two years of additional training 
either in a surgical specialty or in general surgery. 

-. T ;‘ During the past few years we have started shortening training. 

Our Group II Program will be phased out as the three -year or Type II 
residencies are being discontinued, and as of June 30, 1968 the Con- 
ference Committee ‘ceased entertaining any applications for new Type II 
surgical training programs. As of June 30, 1972 all of the three -year 
residencies in existence will lose their accreditation. A number of 
them, by joining with other hospitals, will become four-year Type I 
residencie s. Others Will just‘cii s appear. In 1 951 forty pe r cent of our 
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candidates were Group II candidates,' having trained in Type II surgical 
residencies. Now only nine per cent are Group II candidates and in a 
few years those of this classification will gradually disappear ; 

In 1957 our Board added a very important and strong require- 
ment which should have been added years before -- the se nior year 
requirement. This meant the end of the old "pyramidal" Tr system of 
surgical residency, in which only one chief resident of each year's 
crop of junior residents advanced to the final or senior-year of experi- 
ence. Now, the resident must continue through the senior year of the 
residency program in either the Type I or the Type II residency if he 
is to be accepted by the Board. This has enabled the Board to assure 
that every certified surgeon we turn loose upon the public has himself 
done quite a volume of operative surgery in a variety of fields while 
he has been under supervision in training. 

In 1965 our Board made it obligatory for every applicant to 
submit a certified list of the surgical operations he himself had per- 
formed during his training. It was the belief that if enough operative 
work was , not being turned over to the resident during his training we 
could not certify him to the public as a practiced surgeon when he set 
out into private practice. 

This requirement came at a fortunate time. That was the year 
when Medicare and Medicaid were established, and, with other organi- 
zations, we were concerned lest a disappearance of "indigent! 1 patients 
would destroy surgical training in the United States. In 1966 the crop 
of surgical residents completing their four -year training program 
submitted the required operative lists to the Board. Over six hundred 
who had completed the Group I Program had performed an average of 
504 major surgical operations during training. This set a base-line. 

In 1967 it proved that the corresponding group had performed an aver- 
age of 511 major operations each; and in 1968 a larger group had 
performed 523 majors each. To date, then, there has been no evidence 
of Medicare and Medicaid seriously damaging this aspect of surgical 
training. . .. ; ; o : >■ ! v ( ; o.V-- -'i < , v V 

Another great problem for the American Board of Surgery has 
been the increasing number of foreign medical graduates coming to its 
examinations. .• Fifteen or twenty years ago it was a rarity to see a 
foreign medical graduate at one of our examinations. Then they began 
to appear,, frequently and the number and proportion r have increased 
progressively, without a break, for twelve or fifteen years . Of the 
925 new candidates coming to our Part I examinations for the first 
time in December 1968, 36. 8 per cent were foreign medical graduates 
who had come to this country and taken their four or five years of 
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surgical training here. (The remaining 63. 2 per cent are classed as 
"domestic medical graduates, " having been graduates of medical 
schools in the United States or Canada.) 

, Our first Part I examination was given in 1937. After the first 
twenty-six years, in 1962, of all the candidates who had ever taken the 
Part I examination 90. 9 per cent had eventually passed Part I, by 
examination or by re-examinations. 

The first Part II examination was given in 1938, and in 1963, 
after the twenty-six year period, it proved that 91 per cent had passed 
Part II eventually, by examination or by re-examination. However, 
only 80 per cent of the candidates who had taken Part I had eventually 
become certified, and not 82 per cent, since there is a small though 
constant loss .between the Part I examination and certification. A few 
candidates leave the practice of surgery for other practice, or for 
full-time research, and are thus no longer eligible for Part II; an 
occasional candidate is denied admission to Part II because of unethical 
or immoral practices; and almost annually at least one candidate who 
has passed Part I dies prior to his completion of the Part II examina- 
tion. : ' V. ■ 

The figures just given on our 26-year pass rates were the base 
line prior to the recent more rapid increase of foreign medical gradu- 
ates. I am sure that through the years succeeding 1963 this high pass 
rate would not hold; the foreign medical graduates have run too high a 
fail rate in the Part I examination, which is of the multiple -choice 
type. When 13 per cent, 14 per cent or 15 per cent of the United States 
and Canadian graduates failed, over 55 per cent of foreign medical 
graduates were failing Part I the first time they took it. That figure 
has now dropped to 43 per cent, still much higher than for the "domestic 
medical graduates; 

We have been worried over another development the past six 
or eight years. Along with the increasing number of foreign medical 
graduates we have noted a progressively decreasing number of U. S. 
and Canadian applicants . Why fewer United States and Canadian medi- 
cal graduates are now going into general surgery we do not know. It 
happens that the influx of foreign medical graduates has just made up 
for this loss, since our total number of applicants has remained nearly 
constant.' :, Our surgical training programs are only 90 per cent filled, 
so it hasn't developed because' there has not been enough room in the 
approved surgical residencies for both groups. ” Still, we do not know 
the reason this decrease in the domestic applicants has developed. 
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We also ask ourselves, "How many general surgeons. do we 
need in the United States?" At a recent committee meeting one of our 
own Members came up with the estimate that, based upion the propor- 
tion of major surgical operations done in this country by general 
practitioners and by osteopaths, he and some others estimate that we 
need one general surgeon for each ten thousand of population. Visiting 
was a representative from the Royal College of Fhysicians and Surgeons 
of Canada who pointed out that in a recent survey made in Canada they 
had decided that they needed in Canada one general surgeon for each 
nineteen thousand population. Osteopaths are not licensed in Canada, 
but the C anadian surgeon estimated that the general pratiti oners per- 
form fully fifty per cent of the surgery done in Canada although they 
are relatively untrained in surgery. 

Our Board is. beginning to take interest in In -training Examina- 
tions. We would have to enlarge our Board greatly if we went into this 
field. 



These, then, are some of the problems we have been encoun- 
tering more foreign medical graduates, fewer domestic medical 
graduates, the matter of assuring sufficient operative experience 
during surgical training, and a number of other problems. Right now 
I believe that most of our Members will welcome the time when we 



can drop the internship year. We are all in favor of shortening gradu- 
ate training in surgery by at least that amount. As soon as the State 
Licensing Boards come around to this I think the internship will be 
dropped as one of our requirements. ... .1 .. tl 

DR. McGUINNESS : I have a couple of brief comments and one 
question about the foreign medical graduates. It is my chief area of 
concern as most of you know. Fifty per cent of the foreign medical 
graduates pass the ECF MG examination at the level that it will be 
passed by the lowest twenty per cent of the United States graduates, 
and about a quarter of the foreign me die al gr aduates pa s s the ECFMG 
examination with a score of 75 which is the level at which the 97th man 



out of 100 United States graduates would pass the. examination. That 
is, number one. . Secondly, hospitals in the United States have the dis - 
,, cretion to decide whether in their opinion a given applicant for a . 
residency needs first to take a United States internship,, or whether 
he can be .admitted directly to a residency. It is our strong feeling 
and; our strong recommendation to every foreign medical graduate 
who, consults us that he first take a United States internship.; -But we 
know that: many ; hospitals; admit 

ing without a United State s irite rnship, so that many hospitals approved 
for residency training in surgery are getting people who academically 
are at u substantially lower level than the United States graduate as'. 





evidenced by this one measurement let alone all of our other cultural 
and other problems that we don't evaluate. 

Since a great many of them go right into the residency, I think 
it is quite remarkable that they do as well in your examination as you 
say they do. 

My only question is do you accept a year of straight internship 
in surgery in lieu of the fitfst year of the surgical residency? 



DR. MOORE: The answer is "No, ", a straight internship will 
not substitute for the year of internship following graduation insofar 
as our requirements are concerned. 

However, our Board does accept foreign internship in the case 
of a foreign medical graduate. The reason we have done this is the 
five-year limit on his Visa under the Exchange Visitors Program. 



DR. McGUINNESS: You see, the foreign interns hipi in the 
majority of cases is probably not nearly as applicable as the United 
States surgery or clinical surgery. 

DR. MOORE: The other reason we would like to drop the 
internship requirement is that for fifteen /ears most of the people on 
our Board have thought that the internship year had become more and 
more a wasted year for the individual who is then to take a four-year 
surgical residency, and that we wouldn't be requiring our Candidates 
to take it except that the State licensing boards have required an 
internship. ' v 

CHAIRMAN HALL: We are going to have to change the sequence 
today s o we can have the advantage of the As s ociation of American 
Medical Colleges. Dr. McNulty has Other demanding phenomena this 
afternoon. I wonder if Dr. McNulty could give us the AAMC's view 
relating to evaluation in graduate medicine. 



, ri- : r MR> MATT HEW F. McNULTY,: JR. : I would offer three obser- 
vations . I would certainly urge the introduction of public members into 
the education evaluation process. ; That gets to be a rather glib term 
nowadays but if you look at the medical family -- of which I include 
myself as one -- and say that we exist by and large to serve society, 
then, I think society is entitled to participate ill decisions as to hour 
we are gbing to determine the qualifications;’ the performance, and 
the attainment of those who are going to serve. Introduction of public 
representatives into the evaluation process without destroying profes- 
sional peer re view will riot be easy, and there: are a number of "pro's 
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and con's" that can very well be debated. However, I, for one, would 
urge it. 

I would also urge development of some method by which we 
could evaluate both the goals of education and the goals of patient care. 
We have so much trouble reconciling them, and, perhaps, some part 
of the reconciliation might evolve from careful scrutiny of the goals 
of both of these type activities. Finally, I have been groping, as we 
have been sitting here, for jsome enlargement of this exercise. How 
do we get it to many other institutions that represent what are called 
teaching hospitals, meaning the Council of Teaching Hospitals ? How 
do we get any sort of evaluation into both the institutional in-service 
and the practitioner continuing educational programs that are now 
receiving so much emphasis ? 

Now a word on organizational evaluation. Your emphasis is 
educational evaluation, but I think you would want to know that the 
AAMC has existed since about 1876, and, after that long period of 
time, has completed an evaluation of itself and has evolved changes 
as a result of that look. Those changes, are now coming into being, 
in terms of organizational elements of the AAMC, as opposed to one 
element of organization before. Now there is a Council of Deans, a 
Council of Academic Societies, and a Council of Teaching Hospitals. 

The Council of Academic Societies is looking at graduate medi- 
cal education as it relates to the university. The Council of Teaching 
Hospitals, and a recently, formed Federation of Allied Health Science 
Activities, will provide an umbrella, hopefully, for all of health edu- 
cation, so that there can be accomplished a forum for representation, 
discussion, debate; and, hopefully, leadership and action positions. 

The AAMC is engaged also in the accreditation process with 
the AMA, through the mechanism of the Liaison Committee, which is 
an authorized representative of the _ National Accreditation Council. 

They are also engaged in urging some type of commission, 
council or body of interested groups to constantly determine, evalu- 
ate, and recommend national position concerning foreign medical 
graduates. ' ■ . " •" 

' One comment about the role of the university and graduate 
medical education as related to the AAMC conference in Washington 
on October 2nd to the 5th, 1968. I think it is too early to determine 
any specific trend. . There are positions and purposes related to many 
vested interest groups. I am not sure that a movement to universities 
or a movement from universities to hospital-based graduate medical 
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education or a combination of both necessarily constitutes a threat to 
any of the existing activities. I think there can be an evolutionary 
process here in which the goals and roles and ambitions and interests 
of all can be to some degree preserved. Perhaps education at the 
internship and residency level can be put in the same institutional 
framework as is undergraduate medical education, utilizing the strengths 
of all elements. 



I don't see, parenthetically, any particular disinterest on the 
part of university presidents by their failure to show up at this meet- 
ing. I think anyone who has been with a university president in any 
capacity for a couple of months would quickly realize they are not out 
looking for work. If this meeting involved thie possibility of more 
responsibility for the university, this would be about the last place 
a university president would want to show up. Somehow we need to 
create the exercise model so as to provide intellectual, organizational, 
and university-goal-setting stimulation to attract university presidents. 

I think; if he can be challenged as to the educational benefit to 
the society for this sort of organizational and management arrange- 
ment, and also be given reasonable assurance that the resources 
necessary for the undertaking will be forthcoming; then I believe we 
will get a responsive reaction. I think we will, also, from the teach- 
ing hospitals. Many might consider themselves threatened at this 
point in time, but I don't think that necessarily follows. 

I wanted to close by indicating another educational activity of 
the AAMC, and that is the workshop on medical school curriculum. 

I would just quickly read the conclusions of this particular endeavor 
which was held in Atlanta from September 18 through 22. 



The recommendations first: 



"The overriding recommendation of the workshop is that 
medical schools must now.actively revise the content and 
method used in the total span of the education of the physician 
so that his professional competence will be most relevant to 
the meeting the changing of needs of the people. " 



Rather universal, but I think it was intentionally meant that 
way. The words were chosen with emphasis, and, hopefully, put 
together to accomplish an action orientation. ' V . 



Then, the recommendation is in more specifics: 

"Medical schools must increase their output of physicians. 
All schools should immediately increase the number of entering 
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students, accelerating expansion by redistributing tempor- 
arily the use of existing resources. Medical schools must 
admit increased numbers of students from geographic areas, 
economic backgrounds and ethnic groups that are not now 
adequately represented or that are now inadequately 
represented. 

"(3) Medical schools must individualize the identity of 
the physician to meet /the student's varying rates of achieve- 
ment, various educational backgrounds and different career 
goals. 

"(4) Medical school curricula should be developed by 
inter -developmental groups that include participation of the 
students. Curricula should be ratified by the faculty as a 
body rather than by individual departmental chairmen." 



A slight attempt at humor. Eradicating fiefdoms is going to be 
a very interesting endeavor, but this seems to be most desirable and 
most necessary. 

"(5) The medical school must now assume the responsi- 
bility for education and research in the organization and 
delivery of health services. " 

Let me quickly enumerate some of the findings. Many of you 
were participants in Washington, but just, perhaps, for the record, 

I will quickly read them. 



Duration in years: A trend toward shortening of the 
period between high school and the award of the Doctor Degree 
is present, though small. Nineteen of the one hundred and 
one schools that were evaluated as part of this process indi- 
cated that this was a significant part of their curriculum 
change, the shortest time being six years. Two schools 
planned to extend the duration of medical education for 
: selected students ^ \ v.,,...,:..., , 

"Duration of each year in weeks: Each academic year 
tends to be longer, become longer, when the schools shift 
from the busine ss to new curriculum. . The first two aca -' 
demic years fewer schools request thirty- two week ; k \ .y ; ; . 
calendars, , or more were longer. " The assigned time ;y % . 
fraction, the rigid assignment of course time throughout 
the medical school curriculum is giving way to an invested //, 
amount of free time: arid" elective ;actiyiiy. ^ 
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"The content of the curriculum: The traditional courses 
of anatomy, biology, physiology, etc. , and the clinical ser- 
vices are all being reduced in terms of the total number 
taught. The drop from the basic science courses- are about 
25%; in the clinical, 10% to 15%. Part of this released time 
will be used for new basic science courses; molecular 
biology has been added in thirty-four institutions. Human 
genetics, in twenty-five institutions. Biomathematics is a 
new course in eighteen schools. 

"The teaching format: The rise of the inter-disciplinary 
teaching and the increased integration between the under- 
graduate arid graduate levels are accelerating developments. 
Teaching facilities and materials, individualism, early 
patient contacts, grading and evaluation, the change process 
and many other areas of examination arid change were all 
findings from the study. " 

CHAIRMAN HALL: In development of the relative in-service 
educational program for graduate education, based on your experience 
and background in hospitals, would hospitals and medical schools be 
willing to foot the bill for this costly endeavor of development, of good 
measuring techniques ? ;V ' ; ■■■■■' 



If these techniques are to relate to the practice of medicine in 
a relative sense arid to the practice and its future needs, it is going to 
be very costly to survey the present state of the art/ survey the per- 
formance of people, and, then, how our educational programs should 
be structured and *our evaluation procedure structured. 

: • 1 v ’ MR. McNULTY: Jack, generically, I would respond "yes. " 
Specifically, I think here you get into the tradeoffs or priorities related 
to resource in which we all deal. There is a finite amount of manpower 
space, equipment arid money . '■<> 



So, • therefore, the- tradeoffs are: How do we utilize this finite 
amount of resources at this given point in time; for protracted periods 



say, twelve years; and in the short-range projection, usually, three 



: I would think evaluation is a mechariism: by which you determine 
whether the use of the resource s is being productive ; therefore, in my 
opinion, ’ any "evaluatiori" proposal would have a high priority for con- 
siderati'ori. Sin the however^ what projects should be 

given what rank order by what group of vested-interest participants 



for a finite number of resources ? 
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DR. FRE YM ANN:, I can pick that up as a former hospital 
administrator whose scars are not healed yet. It seems to me that 
you are talking about evaluation. We ought to look really at what 
hospitals are really doing, and not confuse activity with achievement. 

You look at hospital statistics, and you know how many patients 
came in the door and how many go out, but you know damn little of 
what was achieved. I think evaluation of patient care is a very legiti- 
mate use for operating income of hospitals, and I think the evaluation 
of education is just a spinoff of this. 



DR. ANGELIDES: How can you evaluate and commit funds on 
a cost-effective basis. Matt, if you don't first evaluate the problem 
and nave some idea of what you are being c ommitted to accomplish ? 

: ;; i . ; ,We are doing it by the seat of pur pants now, but I wonder 
Whether or not you can avoid e, valuation of what you are doing . >;• The re 
is a finite amount of money, and it has to be allocatedi hopefully, :• 
effectively. r rib} ' .ji' r> .nil;* /&.' 

<■ MR. ; McNULTY: Let me put it\,_, very practical terms. With 
this finite amount of money, the competing interests get to be very 
pragmatic forces. You find the intellectual benefits and the hard 
operational benefits that can come from an evaluation process with 
surgery, medicine, and other disciplines saying, -"But there are a. ,: 
thousand more patients at the front; door,; and they must have this 
: refined type of equipment, which is going to take this additional , j 
amount of space and this additional amount of money for manpower.:" 

This is what I mean by the tradeoffs, and I think we need a 
platform fr om which we. indie ate the benefits thatjean come from evalu- 
ation and, indicate that they are just as useful to society as taking care 
of the thousand additional patients. But they can't be negative, they 
can|t be, , n We are not doing this when we, should be doing it.; 
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vr j ; f C HAIR MAN HALL : J : I think our educati onal institutions have ? 
been ,te sting what the ne eds b>f the teachers were, much more often n 
than] testing the needs of the' studentsk'^s we^ go into graduate educa- 
tion, vthis is very .critical to us> :f^ are; testing our educatipnal ;i 
procedure relating to, the- needs of;the: patie nt. ^As you ^pointed out to 
us, vwe are : a service professioni^^All of ;the health professions; are. 
service professions, and we must address ourselves to this as web:-, 
set up our objectives and evaluation procedure of these objectives. 
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brbii -y? MR. McNULTY:'; You ; ha ve an a dd iti ona l a lly now in society^; 
and thie activist ,student who,v; I 'think, wants some evaluation of what; 








it is you are giving him and is it of any value and is it going to be 
useful to society for the first year, the fifth year, the tenth year, the 
twentieth year.' ■ 




CHAIRMAN HALL: If you doubt this, did you read the editorial 
in the Student A merican Medical As sociation a few months ago ? The 
students observed that many of the curriculur adjustments the faculty 
was engaging in were; as in the practice Of chiropractic, making 
adjustments without knowledge of the cause. 

DR. STEARNS: At the risk of repetition, can I interpret what 
you said to mean that the policy of the AAMC is that graduate medical 
education shall be in the future in the province of the medical school 
arid uriiversity hospital ? f 

'i '-vi; MR. McNULTY : I don't think any policy has come out that 
clearly defined. T think at this point in time there is a clearly defined 
consensus that the existence of internship, residency* and fellowship 
graduate medical education ought to be carefully examined,* and that 
current examination as conducted to date describes many benefits for 
the locatiori of this activity within some form of university aegis. 

fc^We may be talking of a ten-year or fifty-year evolution. That 
part isn't clear^ I think there is a slight trend for the institutions that 
are members of our Council -- 345 of them, which represent about 
23 per cent of the acrite beds in the country -- there is a slight trend 
for those that have been long free-standing to become associated with 
universities.' : -T don't think this trend follows any position that the V; : 
AAMC has taken. -f!.- : ' • 




;j DR i,? STEARNS': I think this is fundamental issue of concern 

$ todayP Is there ; any reaction to : that from the AM A or the AHA? 
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Y/y DR. MIXTER :Y I have a couple of farits that stem front some of 
the comments on the numbers of orthopaedic programs, the number of 
vV general surgicalYprbgrariis, which are uriaffiiiatedi^ I have the "bible " 
here; : MedicaL Education iri the United States^ which iriost of you are 
familiar withiS I pinched Some numbe rs out of it j W-T o ? start in the ; 

Y middle^ ; almost exactly half ; Of the programs in general- surgery are Y . 
university' affiliated or ■ medicaLschool affiliated. Y|To go to -one extreme, 
approximately one iri four of the general practice programs are medical 
sc hook 




quarters^- and- [the neurologic:- program is; .precisely" 5 90 per ^ceri# v . 
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Those are just representative of the spread of residency 
programs which are or are not medical school affiliated. 

DR. STEARNS: These are factual representations of what may 
be a reflection of necessity. , The question is what is the policy? How 
does the AMA react to this policywise? . 

; , DR. MIXTER: I think Jack Nunemaker is going to talk about 

...this. . ;• . ... • . r , ... r • • ...', ; r . : /v . ... ' 

DR. JOHN C. NUNEMAKER: Not unless I am asked to. I 
don’t think there is going to be time. I don’t believe the AMA has a 
policy on that. . .... '• - r ' r>x h. 

>3 We received a Millis Committee recommendation which l am 
sure you are aware of. In some of our personal speculations,: Bill 
Ruhe has said it doesn’t strain his imagination too much to suppose 
that in the next generation or two — after we are all gone -- we might 
not have any specialty boards, but that university degrees would desig- 
nate specialties. There wouldn't be a need for a specialty board. ^ . 



relate to a university and be recognized by some advanced degree. 

This is just idle chatter. We don't have a policy that something should 

be thusi.,and'.so:.....^^.s^-'4 : >o^-- i ;i kt} : < k- 4 * 



; about we; use as our annual 
.reflection of trends. We go back over the last ten years, look at the 
ebb and flow;. You will get an idea of what is happening, and the best 
we think we ca.n do is show people what has happened and let them 



draw conclusions . 



;. " if.' 



DR. MIXTER: If you project the curve that we know now is 
the continuing decline of independent programs and continuing increase 
in affiliated; programs, you can .*t make a straight line prediction, but 
it looks like , it would take, anyways, fifty years at the present rate; 

DR. STEARNS: , To have all programs affiliated? 

• ; DR. MIXTER: Yes, something like that, and, furthermore, 

the universityyiaSjMatt pqinted^outf^-vthe universities aren^t in a 
positions to take, on.much more than they have got 

^ STEARNS:, ; On the other; hand, Vif the^trend for the univer- 
sity to take on more -and: more affiliation with community hospitals -r y : 
who represent the other spectrum with the training of undergraduates 
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and extended into graduate education -- it may happen an awful lot 
faster than fifty years, like maybe five. 

DR. MIXTER: That is what I say. You can't predict a straight 
line because there are forces involved here that weren't operative in the 
past, and, so, any prediction we make from past trends is rediculous. 

DRV STEARNS: But the forces can be moved by thinking in 
terms of what will potentially yield the desired result, which is, maybe, 
more medical students and more trained surgeons and more trained 
orthopods in a combined effort between university arid community hos- 
pital utilizing existing resources for expansion. George/ I am sure, 
is concerned with this and others, too, and there is nothing wrong with 
making policy if it looks like a good thing. I hear Matt McNulty saying 
this is what the medical schools may think is a good idea, because 
they have a better way of controlling things. I hear you saying this is 
the’ trend, •' anywayV'Y •. Y •. % 4 ;-4uy 

■■■■‘"y I don't know what the American Hospital Association says. I 
think our Association would probably say it is a good thing. If every- 
body says it is a good thing, maybe it shouldn't take fifty years. 

Maybe' it should take fiveV ; V ; V ; ‘-v 

l r CHAIRMAN HALL: I can't constrain myself not to comment 
at this point. I think I would agree with you if the association is a 
reciprocal one. It is a merging of the academic excellence with the 
demands and needs of the population for health care. 

c: T 'think that AHME represents better than fifty per cent of the 

practicing physicians or fifty per cent of the practicing physicians are 
associated with institutions who have directors of medical education 
of our organization; 

’■fj Y;DRV HUBBARD : Mr! McNulty, I find it very difficult to under- 
stand why the hospitals of the country -- and I wilk say the hospitals, 

I won't cite your Councir or the AHA or the AMA — but why the hospi- 
tals of the country are doing virtually nothing, to learn more about the 
competence of a large group of interns and residents serving within 
their walls. I am referring, of course, to the foreign graduates. 

/y i I i' sugge sted i as t night that maybe " some national program might : 
be mounted to- get information as to the competence of this large seg- 

American 
lead 

ballbonV .'■‘but- would ; ybu ^care^tb' cpmm 

80 y&x Yi : 4 Y 



frs:. 




I 

I 

I 

I 

I 

I 

I 

I 

I 

I 



I 

I 









w^grogB B mw ia grawSKHam g M^ 



MR. McNULTY: Over approximately, perhaps, the last six 
years a great deal of the leadership capability in hospitals has been 
distracted from critical inquiry of a number of developing activities 
because of simple, but total, demand on energy and ability to stay 
solvent, stay in existence, adjust to the new forces that have come 
into being in terms of additional beneficiaries in society, additional 
demands in society, and an activist type of student group and public 
group. Briefly, overwhelming societal demand and need has precluded 
many other desirable endeavors. : 

Your next question might then be simple: That is, if you would 
eliminate these forces would these hospitals immediately jump to the 
opportunity that you indicate ? I think not, and I think there the chal- 
lenge is to groups of this type, to the AHA, to the AMA, to the AAMC, 
to your group which has such a repository or expertise and know-how. 
We have to get their attention, and, then, once we have gotten their 
attention, we can move on. How do we. get their attention? Just 
through the time worn system of keeping working at it. However, I 
would add that an AAMC -sponsored endeavor in this area during 1966- 
67 will be issued shortly by the Harvard Press that represents a 
positive action by AAMC including its Council of Teaching Hospitals. 



: Mr.. Chairman, may I say for the record, I may have expressed 
something so poorly that Norm interpreted this incorrectly, but I would 
want the record to indicate that at this point in time the AAMC is in the 
same position as the AMA. We do not have a position. ; 

v- 'iDR. GEORGE MILLER: The implication seems clear, on all 
sides that the desirability is moving graduate training into the orbit of 
the university. While this may have. a universal appeal, I would hope 
that if it were brought about, then, the university would begin to exer- 
cise some of the spirit of critical inquiry it is supposed to represent 
in looking at those educational programs. The only systematic study 
of the internship, for example, was done of university hospitals, the 
Saunders Study.;. I don't think one could be very happy about the findings 
that were brought forth the re . .Yet, the re sponsivene s s ( of the university 
to that study has. been almost non-rexistent as far as I can tell. ; r :: 
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Sickness and Society may not be a popular document in university 
circles, but it is one of the first systematic studies of this delivery 
system in the context of a. university hospital^^;v%- : ; - ;v :v 






We happen to be moving Out into the community now as a uni- 
versity. My hope is that we can do a study comparable to Sickness 
and Society in some of those community hospitals before the university 
moves in and after to see whether we improve, fail to alter or, in fact, 
to do harm to this system that we are talking about. 



DR. McGUINNESS: The very existence of the organization 
which is sponsoring this conference is clear evidence of the interest 
of hospitals in doing something about the education of their staff. I 
was a trustee of a hospital seven years ago which at that time no more 
would have considered putting out twenty-five thousand dollars a year 
plus benefits for a Director of Medical Education than anything. Four 
years later they did exactly that for the very reasons, the very things 
we are talking about. I repeat, the existence of this strong organiza- 
tion today is a result of the recognition on the part of the Boards of 
Trustees of hospitals that something has to be done about it, and it 
represents a tremendous investment on the part of trustees of hospitals 

DR. NUNEMAKER: I could clarify one thing I said about the 
AM A position. The House of Delegates did act, and it did include 
universities exerting more leadership and developing better material, 
etc . So,' they did recognize the responsibility of universities, but it 
did not take it to' be acted- upon, ' 



i: 1 • •' ^ M. DR. STEARNS : 
arrangement. 



The takeover is not 



liT-y. 



a cooperative 



DR. ROBERT L. EVANS: There is one thing we have been 
talking about, the real world; and you can talk all you wish about 
physicians; global responsibilities and trends . The fact of the matter 
is — to carry Jack’s analogy one step further as we did a couple of 
years ago in one of our meetings -- the motivating force in our nation 
and this economy is cash, and there is no doubt at this point that the 
cash delivery : is in terms of research and medical care and in terms 
of moving the university out into the community, and it is going to 
respond to this cash. I hope we can evaluate the response as it occurs. 

‘ DR. ANGELIDES: If you accept the premise that education 

does something, iri this case to take care of patients;^ that education 
is a step towards accomplishing this goal;^you cannot isolate continu- 
ing education. v I- think this argument as to whether the university 
-should get involved or riot is^ridiculous. V It obviously has to. You 1 •• 
can't take the tail end- — eoritinuing educatibn -- and say we cari solve 
these problems; somehow or other. You have to start going back like 
Ariadne in the labyrinth, ^because 5 of the Jeduerition^^ 
arid ‘riot stophiritil you are back to the public school system;' , I think 
we are making artificial:‘distinction's^ari.'dfeworir-ying/-about-:uirireal'-pres-^ 
sures. It is apparent you cannot separate these elements;; : 



CHAIRMAN HALL: We are starting off the afternoon with a 
discussion of the evaluation procedures as viewed by the American 
Academy of General Practice. 

Carroll Witten will give the AAGP's point of view. 



DR. WITTEN: The strength of the American Academy of 
General Practice has been only one thing; and that is its requirement 



zation. We are the only national medical organization that does require 
readmission to membership by, certification of haying completed a 
number of post-graduate hours, namely, one hundred fifty, during a 
three-year period. This must be done in order for membership to be 
maintained in our organization;; and it strengthens the requirement 
for reelection to membership. : ''.y- n yfeiv: ,xU .5 } Ay;r. ;,.0 .* o' 



This, I would recommend to all organizations, and I believe 
that some day many other specialty groups -- as ours is a specialty 
group — will adopt this posture, ; r v <;.! ; vy :,b ■, kk.-h ■ :-.i h- 

I don't mean to say that this has worked one hundred per cent 
successfully, but, in general, it has been a success. Has the benefit 
been, measurable as to those who have maintained their membership ? 
We think yes> and there are some statistics that would prove pur 
point, although, admittedly, it is a difficult thing to evaluate. ; y; 

y, But I don't want to talk about our past so much as to tell you 
about what we plan for the future in order that you might understand 
what our hopes are. 

;/ r>] >v ; T would teir you this. ;v I was a prisoner of war in World War II, 
not as a doctor. I flew bombers, V was shot down ove r Germany, and 
became somewhat of a troublemaker as a young Air Force officer. I 
oftensspent time in? solitary. r? V ."-el 



y | One time I had been in solitary for three weeks, and after 
coming out I was sent to a maximum security prison in Nuremberg, 
the same -prison where they held the War Grimes Trials. 

. . < , • There were four other men who were on this .maximum security 
ward. y We. were-all in our separate cells,»y but we were near one another 
There was a Serbian general, a French resistance worker; ( a Russian 
soldier who I later found out was a corporal, and a German Jew who 
was a deaf mute. r And L was there, as a young Air Force major. 






None of US' spoke the other man's language, and, of course, 
the German Jew, the deaf mute, didn't hear or speak at all. But in 
* two weeks we learned to communicate. Within two weeks we were 

able to transfer information from one individual to another. In 
American medicine, they haven't been able to do this yet. 

And that, to me, is the value of this meeting' because for the 
first time in any meeting that I have been in — and God knows there 
are a number of them this is the first time I have seen free and 
frank discussion 5 between' the Various medical specialties, and, per- 
haps, some recognition of the problems of the other individuals in 
other fields. 1 Maybe we have. begun to learn to communicate. > ! ■ 



Now, the general practitioner has often felt like the one child 
of the Catholic couple that had twenty children. They baptized nine- 
teen, and they kept one for control. 






This, to us, is what most of you in some of the medical 
specialties have done to general practice for some few years.' We are 
tired of being a control, and we propose we become the twentieth bap- 
tized child tonight at the Liaison Committee Meeting: 5 - , 

We' have submitted 5 our filial application for the certifying board, 
which, in our opinion, will bring about quality examination of the candi- 
dates, quality training programs, quality individuals who enter the 
field, and produce a product for the purposes .of meeting the needs of 
the American people. : That is, a family physician to whom they can 
turn for’ their prin Vry medical needs. V s -- \ts XV ;• uv4u„-:- ■ 

We propose these points: A three -year training program after 
• graduation from medical school designed tb produce family physicians; 
a three-year training program that will encompass the essentials that 
. were just approved by the: House; of Delegates of the American Medical 
Association after being submitted by the GouriciTon Medical Education 
’■ this past December. V i vi ^ ^ jt;; : -’SV V V y 
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We propose that: at the end of this three -year training program; 
that the individuals will be tested by a quality examination prepare din 
conjunction with the National Board of Medical; Examiners in Philadel- 
:; v phia. : ; We- have' -been working with them now for ’ almost two. and one -half 
year s and 'weha ve the examination re ady to go . > This examination i s 
? - two full day s in le ngth'.V T he r e ’ ar e, s ix hours of testing on each day. 

have sixihours of testing, primarily that most identified 
- with Part II of the National Board type of examination questions, the 
multiple choice type. V I might add .that we dori ' t ge t into the que stions 
that we think are absolutely superfluous to the practice of medicine . 

V V • : V . ; ;X : ' 
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We get into primary practicing management problems dealing 
with patients and how one reaches certain conclusions and what con- 
clusions one does reach. We also have deliberately avoided some of 
the pitfalls we think occur in some examinations, such as trying to 
recall what appears to us to be superfluous information, such as 
esoteric normals of certain tests. In these more unusual tests, we 
present not only the tests, but the normals that are tested in that 
particular institution that the individual might be using. 

In addition to this type of question, we are using the movie 
film. We have eight films that are part of the exam. These are short. 
These are silent. These are to test powers of observation of the indi- 
vidual, and they are most excellent. 

We have, also, programmed testing questions, the type of 
question in practice management where one has a problem presented. 

He rates the answer right, wrong or equivocal, and moves on from 
that to another answer, and, thus, he has pluses and minuses charged 
to him and credited to him in order to derive the score. 

These examination questions, and this lengthy comprehensive 
quality examination has been prepared by ten members of the American 
Academy of General Practice, Eight of us hold academic appointments, 
teaching at various schools, universities, in the country. Some of us 
hold multiple or duplicate appointments in several medical fields. 

In addition, all of these ten men are in the private practice of 
medicine, and, therefore, have a feel for I heard the statement made 
several times today -- for the real people and the real doctors. • 

We are, if anything, the real doctors, although we seem often 
to be separated from reality. . 

’• HALL : C ar r oil, could I start out the questioning 
by asking the question, has the AAGP or the AAFP defined its universe 
in the sense that Rosinski or Elling defined universe for us ? 



//.y; This is an extremely difficult thing to answer. 

Jack, because all of us think in different terms as to what we mean by 
this, but, yes/'iri'some ways we think we have. ^ This is the same type 
of question that has been posed to us for many years. In our travels 
" V around the country arid the me dical schools the dean or the professor 
of medicine would say, "Tell us what a family physician is, and we 
will be able to produce them, but until you can identify this, we can't. " 
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Well, we worked on this for many years, and, as you know, 
after a six-year effort, we came up with what is called the "Core . 
Content to Family Medicine. " It is certainly not a perfect document, 
but it is, most assuredly, a step forward. It is one, I might add, that 
might be attempted by other medical specialties to clearly identify 
their universe as well as we ha.ve tried ours. 

DR. ROSINSKI : How did you decide on particular techniques 
of evaluation? 



DR. WITTEN: Although eight of the ten of us have university 
appointments and have jpar tic ipated, of course, in designing some 
type of examinations for students, we were all rank amateurs. We 
leaned heavily upon the National Board of Medical Examiners for 
their advice, and they have been most cooperative. , . . 



Their people are experts. We listened to them, we read the 
“ books, we studied. We think we have learned our lessons. We went 
through several hours of heavy and hot discussions, and, finally, we 
came to some rough outlines. From these outlines, we have been 
able to decide the type of examination question that would adequately 
meet our needs . it sort of has been trial and error. • 

/: J - - • 1 We have, incidentally, had an opportunity to show this exam to 
repre sentative s of the American Board of Surgery, of the American 
Board of Psychiatry and Neurology, American Board of Pediatrics, 
American Board of Internal Medicine, and in each instance, these 
men have all expressed to us that they believe it to be a quality 
examination;.' v; ’ ' ; “i v - -v^ •' y '••• 



DR. GREGORY:' At the outset, you ‘identified one of the strengths 
of the American Academy of General Practice, and that was the require- 
ment that its members produce evidence of continuing education before 
they could be reelected at the end’ of ‘a three -year period.:.;', Do you pro- 
: pose to abandon this requirement for readmission? •, 

DR. WITTEN: No, sir We do not plan to abandon it. We 

plan to continue it and, perhaps, to strength^ it in a way I will men- 

tiori in just a minute. We plan' to continue it by a requirement in this 

of recertification, one that' I know raises some eyebrows, 

but one that you' have heard usipropose before and others have proposed 

before us. r \- ;; «/:• ■ .••••-'■ •'••• •: 
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/“ W e have in our final application a definite state ment that we 

intend to reexamine and recertify on a six-year basis. This would be 
the same as two, three -year memberships in the American Academy 
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of General Practice, and the requirements for reexamination would 
be maintenance of the same three hundred hours of acceptable post- 
graduate study during a six-year period in order to be reexamined. 

There is one other proposal, and that is one, incidentally, I 
would offer to the Board of Surgery and the American College of 
Physicians and others who are in the process of thinking about setting 
up some self-evaluation examinations. 

I think the time has come when voluntary membership in an 
organization such as ours and the American College of Surgeons and 
others should require some voluntary self-evaluation examinations in 
order to maintain membership. In other words, I think the College of 
Physicians is right in what Hugh said about the way they have done 
their examination. I think it is fine to keep it confidential. I think it 
is fine for it to be done exactly as it is except for one thing. , I think 
they should require that it be done by their members in order to main- 
tain membership in the American College of Physic fans and done every 
five years or six years or three years, whatever period they decide. 
This is the way, in my opinion, we will keep other people from setting 
up recertification requirements for us. which will actually be relicen- 
sure requirements. • : 'm- '■■■; p:.-*.* Vb-V-i ' ?. 

■ ■ DR. FREYMANN: Carroll, first of all, I would like to con- 
gratulate the Academy, as I have on other occasions, for the courage 
in taking the stand on recertification, but I am surprised you would 
still continue the requirement to show so many hours of postgraduate 
education, because now this is going on the assumption that your 
examination really tests. What difference does it make if a man puts 
v in a thousand hours or no demonstrable hours? . 

DR. WITTEN: I think that there is a lot of validity to doing 
away with the mandatory hundred and fifty hours once the recertifica- 
v v s tipn examination has been instituted, but until this ; transition period 
vis over ^ •• Lbelie ve we must maintain the American Academy of General 
Practice as an /entity. i :- •• 

?;fev'DR v FREYMANN: Perhaps I misunderstood you. Once your 
recertification exam gets going, you probably will abandon that arbi- 
trary requirement. 

, iV- DR. : WITTEN: I belieye we will abandon it for those individuals 
.who have their; certification as members of the American Board of 
Family- Practice . I do not believe it will be abandoned for those who 
• ./£ do not take .the 5 certification examinations . vf? l ■/ 












DR; PERKINS: Dr. Witten, will you tell us briefly what are 
your criteria? What will be the qualifying criteria for admission to 
the Board examination ? ! ‘ '• • '■.•••'• ; ’ 1 1 

DR. WITTEN: We anticipate there are two categories of people 
eligible to take the examination. We set those up this way. One cate- 
gory are what we call the residency eligible candidates. These are 
individuals who have completed a three -year training program which 
has been approved by the Council on Medical Education of the AMA in 
the field of family practice or general practice in the past or can be 
interpreted to have been approved at that time by the Council. We 
don't make this interpretation. They will have completed, then, a 
: residency program. ^ r y[ 



: ; : ’ ::r The other group of candidate s are those who have been in prac - 
tice or who have taught in medical schools or, I might add, taught any 
" Mother place actively in the medical profession, which would coyer the 
teaching hospitals that do not have university affiliation. They must 
have taught actively for a period of six years or been in active prac - 
Office for' six years and have maintained the membership requirements 
for the American Academy of General- Practice, and have met, then, 

• the three hundred hour requirement. y ; ! 

: I would point our here that a man does not have to be a member 
of the American Academy of General Practice, but he must show evi- 
dence that he has' had this three hundred hours of postgraduate training. 
He does not have to be a member of the American Academy of General 
Practice in order to be examined. ■ #0 ?«• . :o.n -i : nv 

DR. ROSINSKT: C arrolljC I am a little confused yet.?- If I inter - 
pret you correctly, are you going to have two parallel groups, then, 
v realiy ---the Academy and the Board ? -XStfi-'r-.;'. 

r K IDRi-' WITTEN:- Yes^ ^ust ; >Iike?.'tKe^ Surgeons 

4'and the American Board of Surgery;r ; Just like the American College of 
Physicians and the American Board of internal Medicine. 

fev 3: DR . ROSIN SKI: ; D 6 you see * any difficulty with two different 









WITTEISfr I think there will be initially. I think that, : 
v- however /‘^jh'st 5 as* today the : quality- standards of the Ainerican College 
of Surgeons is ^dependent upon the American Board of Surgery, I think 
that the Ameficah' Academy 'of General Practice, which has been the : 



only leader in general practice/ will be elevated accordihgly as th 
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American Board of Family Practice is established. I think there has 
to be one political organization and one; educational, qualifying, certi- 
fying organization. . ^ • • ••.■• V i" 'v- 

DR. FREY MANN: No grandfather clauses ? 

DR. WITTEN: N o grandfather clauses. I am glad you mentioned 
that. . : u 



CHAIRMAN HALL: You have a very outstanding opportunity of 
introducing an in-service evaluation procedure now as the new require- 
ments of the family practice programs are being accepted, i Is this 
going to be followed up on? W;; : ' : y if -r-.if.i . ; V: It;.:'-; 

DR. WITTEN: Yes, Jack. We have given serious consideration 
to this. As a matter of fact, I have about a seven-page proposal, that 
I am going to present to our Board of Directors, which calls for the 
establishment of these in-service examinations . I am; confident it will 
be bought by the American Academy of General Practice. This will 
be a requirement throughout the training programs. ■, - i ■ ; 

CHAIRMAN HALL: What is it that has made this group flexible 
enough to explore new ways in evaluation for continuing education and 
in graduate education and development ? •> ;> ; 

DR. WITTEN: When you are the only one out of twenty that 
hasn't been baptized;; when you have been kept for control, maybe , it 
was the best thing that ever happened to us, because it caused us to 
be quite introspective. .It has caused us to decide that the successful 
mechanisms that have been used by other specialties and certifying 
board were not enough for us, because we never had been involved in 
quality educational programs, and , therefore, we had to start off at 
a different -r mind you, I didn't say ''higher " — we had to start off at 
a different level, one that could be understood by people. They could 
see by it that we mean business, and I think self-defense is the real 
reas on that we have done the se things. . / ■ : . V; v •:> -or: / . : ; } • :/ j 



CHAIRMAN HALL: The next discussant will be Dr. Howard, 
representing the Council of the AM A. '$■ •'...•>^•‘7 V >. 

DR. RUTLEDGE HOWARD: We are in AM A really a confedera- 
tion of the other societies, 7 Without them we wouldn't exist, and, 
therefore, we do need to take from the others.;? I feel that the specialty 



boards, the individuals in hospital medical education represented by ; 
your own organization here, and the government thrusts which are 
pushing all of us and making us respond are going to make continuing 



education a much bigger thing. As others wiser' in the field than; 1 1> 
have said, it may make continuing education the biggest educational 
effort in medicine, in terms of pupil numbers, faculty numbers and 
individual programs. 



We have three basic things going on in AM A in the continuing 
education field. We are in a voluntary accreditation system for insti- 
tutions delivering continuing education to physicians. We hope that 
will broaden to the other allied health professions. We have thirty- 
nine institutions now accredited— if you count the first efforts of 
pilot test programs -- thirty-nine including those pilot test ones which 
include about fifty-five per cent of the courses given in continuing edu- 
cation in this country. We are hopeful that we; can combine thoughts, 
and, possibly, forces and effort with the Canadian side in that accredi- 
tation system and obtain as well as give information in this bipartisan 
effort. o ; > ' a >..v : ' 
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We are also in the? position of being a coordinating agency.: 

Very little, I feel, has been done in this respect by AMA. But we 
hope to bring various groups together for interchange of continuing ; 
education. 

brs In the coordinating effort, the AMA, to me, when I was in 
practice, didn't represent very much. I got the first week in August 
issue of JAMA, saw dozens of pages of courses listed, put it in the 
pile with the rest of the journals; I said I would read it next week. 

I did pick courses from other methods , from other informational ; : 
sources;: and, then; checked back and found those Courses were, - 
indeed; 'listed by JAM A;? and if I have only used it, I would have been 
helped ‘morej;;:* . -{>'?• b: :: U- £i.-. -.yucyrc; 

i TV? We are trying not: just to help coordinate continuing education 
by this course listing in the: Journal of the AMA, but, also; by bringing 
together forums -- in the most recent instance, state medical associa- 
tions who are interested in the field. There has been lethargy in our 
component constituents which we hope to correct or, at least, motivate 
into better channels. W e also hope with the new decision of the House 
of Delegates approving, the final details of what we call the "recognition 
of worth" for physicians who participate’ incontinuingeducation, to 
create a little bit of motivation for the individual doctor. 

■V.; {fey This is a voluntary effort to give credit to physicians who do.: 
•participate in programs which very much parallels the Academy of 
General Practice system; ^ )yp;s;:Ay'-. 
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We have admired that system as physicians whether we are in 
the AAGP or not. And from the AAGP system the AMA has borrowed 
much and patterned our system of one hundred fifty hours every three 
years giving recognition to the participating physicians. 

Beyond that I don't see much/ I am too close to it at the end 
of my first year to see the forest because I am still looking at the 
individual trees. • : v . : A 1 "-/ ‘ 

DR. MIXTER: How long have we been publishing those lists ? 

DR. HOWARD: I am not sure, George, I think it has been 
about four or five years that I can recall seeing them. • We now have 
nineteen hundred ahcl thirty -six courses. > ^ " v 

We are hoping to improve that listing and to get many more 
institutions who now give courses to want to be in that listing if our 
recognition award to the physician is to mean anything. : 

We hope that recertification will not be a problem, but we hope 
that the umbrella effect of the AMA recognition award's program will 
take the place of actual recertification. — ’ .■ ‘ • ■ > 1 \ ^ 

^ " I was very interested to hear Dr. Witten's statement that there 

will be recertification for the Board men in the new, we hope. Board 
■ of Family 'Practice;' ?*’••/• >V ; v- 

| DR. MAX MICHAEL: Recognition, is this a carrot to beat off 
the stick? ','• ••• ’ { P vvl ' r r '' iy . ; -M - . 

'••'.v-/-DR : .- HOWARD : It is the lump of sugar, right. ; 

We expect something will come out of this. Not necessarily 
recognition or certification for one 'hundred fifty, hours, but, - probably, 
enforcement of just what Carroll said, a self-assessment for every 
physician in specialties and in general practice. That will probably 
be the eventual shape and pattern of things. ? Whether it is in my life- 
time, I don't know.- • :c : jV; 

DR. f'WINSTON MILLER: Have they planned any kind of con- 
tinuing education based on any kind of performance testing ? 

( : • v ; 4 v DR . HOW ARD : / You mean programming it ourselves ? 

;?f\/pRvV--WIN^ :^;Or: improving it or encouraging it tb 

develop the concept of education based on performance . v/ 
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.... CHAIRMAN HALL: What do you mean "performance, " Winn? 

DR. WINSTON MILLER: Professional performance. I mean 
the quality -- if you like to use the term "quality" of medical care or 
the deficits of knowledge at the level of application. 

DR. HOWARD: Individually, yes. As an organization, I , 
haven't heard any policy statement on it, but individually I have never 
heard anyone say he is against it. It is, almost an assumption at staff 
level and in committee and in council, that this is a goal. 

Now, the methodology, whether it will have a self-assessment 
method or not, I don't know, but we are certainly for it. 

. .r? DR. ; MIXTER: (.For reasons of very practical nature, we 
haven't mounted programs such as Rut is engaged in now, because 
we haven't had the manpower, we haven't had the money. It is just 
that simple. 

-jj. We have thought about it at the staff level. It has been talked 
about by the Council and people have tried to institute an evaluation 
process, and all of those who have tried it have been defeated. We 
.don't have any good yardsticks at the moment. They all turn out to 
be rubber yardsticks, you know, and for the Council to adopt a straight- 
forward goal would infer that we knew how to get there* - and, thus far, 
we don't. 

Does that, more or less, answer your question? ■ 

DR. WINSTON MILLER:,; Yes, except I think that the force 
bringing about evaluation of continuing education — based on appraisal 
of need -- could be implemented even by statements that you could not 
do anything about it yet but you .hope that somebody will do it soon y 
because; it is of vital importance. ^ ^.y v- i • ) *1 ; :-V- j s - 

^ DR. HOWARD: Relating to this, .they did spend about a quarter 

of a million dollars on picking out a pilot area of Utah, and studying it 
with a hope that we might expand it into the different health problems 
to see what improvement in health care delivery for each given problem 
can occur,, by .first establishing the needs in the area of each problem. 

■We have been in touch with .the ? R . ; M; P . .• with^this questionnaire 
of various medical programs they have put out, trying to assess needs, 
first, .and, then, set- up objectives which can be measured as an accom- 
plishment after; they have, tried; to do something. about them. 
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: . ,. DR. WINSTON MILLER: Another question. -Have you con- 
sidered some kind of. endeavor ..to employ behavioral scientists 1 /. 
expertise to define the roles of our now twenty different kinds of 
doctors? .... •. ...... ^ 

DR. HOWARD: We have recently talked with a man interested 
in behavioral sciences, and I think, fairly conversant, with the language 
of them, trying to quantify and quantitate some things, but not with 
regard to a single specialty. ; ; 

DR. MIXTER: It is kind of like fighting a brush fire. The 
various boards and. specialties overlap. You can take the lower jaw, 
for example, and there are five involved. They all think it is their 
own province, and to a certain extent, it is. We fight these brush 
fires, by pouring a little water qn here and there and manage to keep 
them in fairly good relationship to one another. The Council and the 
Review Committees that represent it are constantly attempting to 
define areas, but there is no sirigle coordinated effort. 

One of the things that we are hoping will develop out of the 
plans for a super Commission on Medical Education, if we can ever 
get the various political overtone s to die down, would be just exactly 
this — to define specialty roles. This would be one of. the provinces 
of such a group. It is a very difficult problem to state clearly now, 
because there are so many different plans, so many different sides 
for this supercommission. ... . . . ^ ,. t . 

This is all very much in the works right now. There is a lot 
of jockeying, and, maybe, in-fighting, but the thing is going to come 
about. What form it will take, and how it will be paid for I don't 
believe we know, but we have recognized the need for it. 

DR. GEORGE MILLER: Dr. Howard, for a long time univer- 
sities have been protesting, with increasing vigor, the imposition of 
external standards upon what they are to be responsible for. I wonder 
what it is that leads universities now, particularly in view of the talk 
of their responsibility for graduate education, voluntarily to seek the 
embrace of another accrediting agency, to say that what they are doing 
in continuing education is or is not ac ceptable . ; 

DR. HOWARD: I don't know what motivates the university or 
,, the medical school, but I feel a lot of it is public pressure. I feel 
that the Manpower Commission Report in November of '67 might have 

given- some- thrust., t. v ■ ' : ■v.c.i : > 




DR. GEORGE MILiLER: So, we are saying on the one hand that 
the university must respond to public pressure for accreditation- and 
on the other hand that they, must assume individual and independent 
responsibility for developing programs under their own jurisdiction 
for internship and residency. / 

1 DR. HOWARD: Yes, I think the medical school is being asked 

to take on a lot* more than it ever' expected to even as recently as five 
years ago. As I visit various medical schools with or without univer- 
sity connection, I get the feeling that the medical schools at the moment 
are just giving lip service to continuing education, in many instances. 
They are doing something, but the greater noise we hear is more lip 
service than it is actual accomplishment. ~ f' ' 1 

In graduate education! feel the effort is a Tittle older, but I 
funk they are still groaning’ under the load and the cost. I see most 
of the university and medical schools and catalogues aimed strictly 
at undergraduate s, and thinking it is still the old goal of undergraduate 
education. yV'. 






; ’• ' I don’t know what is forcing them in, but I think it is public 
pres sure; I think it is government, representing the public, and I 
think it is young students I think it is a snowball. : ■ ’■ 

DR. GEORGE MILLER: Almost seems as though they are 
letting themselves in for more pain if they are trying to get away 
from the external accreditation. 



r , u j v DR. WITTEN I; im a little bit' concerned to hear that consi- 
deratibn is being given to the establishment of this superb ommissibn. 
Not that one might not be needed, but this seems to be our problem 
in medicine. As the old saying goes, ''You get two doctors together, 
they will form three organizations so they can all- be officers. " 






This is what the AAMC has done in its establishment of the 
four Councils! This is what the AMA does in its establishment of 






councils, some under the' House of Delegates, some under the Board 
of Trustees; It is what the American Academy of General Practice 
does in its Commission and its Committee when we set up a hew one 
ostensibly for a new job. f;. , v : >'• 
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vf'Cn'V If we think' we are going to solve these problems by establishing 
another commission, a supercommissibn; which has no authority over 
the Council on. Medical Education, or over the AAMC, ; or over the 
specialty boards, or over the Advisory Board for Medical Specialties — 
if we think we are going to do it by setting up another organization 
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instead of getting an identification of the jobs of the ones that we have 
and getting a coprdinating effort between them, then, we are fooling 
ourselves again. That, to me, is what is wrong with the Millis Com- 
mission recommendation, as it is being approached by the AMA and 
by some of the rest pf us in medicine. ; 

We look at it as an opportunity to smooth things over for the • 
moment, but the Millis Commission calls for an entirely different 
thing. .It called for one body which will supersede the Advisory Board, 
the Council on Medical Education and all of the certifying boards in 
this country. This might be exactly what is needed, but it cannot be > 
approached by merely setting it up on paper. 



DR. .GREGORY: I was thinking about the emergence of post- 
graduate education as a university responsibility. .With the /revision 
of curricula, in many schools — at the urging of the AAMC -- a good 
many of the specialties are finding themselves literally squeezed out 
of the curriculum, so far as time is concerned. And as they are 
. squeezed they are told, "This is the kind of thing that ought.to.be done 
in the post-graduate period of training, not in the hard core of either 
■clinical or basic science. " .... 



: ) ?. A' 



Maybe, therefore, they will simply translocate the student one 
notch up fthe scale and, come into the picture of postgraduate education 
of specialties almost by default. h: : s ' W. p jp : /,.■* 

DR. BROWN: I have been concerned ever since the inception 
of this accreditation program for continuing education by the AMA. 

By implication it is suggesting that the programs that aren't inspected 
and, accredited might not .be acceptable . It .would be, a poor choice for 
anyone to , spend any time and effort in other kinds of things. 

: I am, thinking of the things we ha ye done at our own hospital for 
the last four years • — I really don't know what to call them. They cer- 
tainly are not education by the usual standards, but we have made some 
rather, astounding changes, I think, in what doctors do in managing 
patients. But I can't imagine my going to the AMA and saying we want 
to accredit our , 
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• ; -.HnoK Our, audit program, I, guess, ;is sort, of what we vjare, talking 
about.|y To a certain extent other, people are thinking about mpving into 
this area -- survey of quality care and measure to see if physician 
behavior has changed to meet these needs and improving patient care. 

: : : 'vVl-;.;ariVi'a^r a id o. ye said, -"Well, 

I can't imagine that being ari accredited kind of .thing, whatever we call 
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They will say, "But, gee; there is a course down the road at 
the university that is accredited. That is where we will spend our ' 



time . 






Now, I think we have got this problem presently well-solved 
in our own hospital, but I am concerned about a lot of other hospitals 
that might/ hopefully; launch the same kind of pr ogram we have . 

DR. HOWARD: Clem, I am glad you brought that up, because 
being wet behind the ears in the AMA and fairly seasoned in the solo 
family practice, I came into AMA with almost a lone voice. !i : • 

George has been in practice. Very few other professional 
people in the AMA staff have been in private practice 1 . I feel that 
many things are continuing education, and at the moment we are con- 
cerning Ourselves with what we can label in many different ways. 

I don’t think formal courses are necessarily the big effort in 
the continuing education of a physician. I feel the average good doctor, 
even in a specialty, can get an awful lot of refreshment of the old and 
adding of the new knowledge and skills talking with just one good col- 
league, and another one next week, and another one the following week. 
Maybe not in his own institution because you tend to get inbred, hut 
talking, especially/ to the younger men who have been well-trained, 

I feel accreditation implies formal courses/ 1 My natural feeling is . 



that 1 are' less 1 than formal course s — les s in stature ; but not le s s in 
importance, y } . '‘'y'"' ' v'\ /"• •'//'' 

; - ' v c My feeling too, c ; is that most doctors ask for consultations for 
genuine help: ' They are not 'what we c all’; 1 'c ons olatioh consultations " 
as the patient nears St. Peter. I think that serving on record review 
/homihitteies -appears? ^compulsory, but in m hospitals, even little 
ones/- you can or cannot serve, as you wish/ - 1 

§ • I// 1 do 'feel a 16t of rthe se tMngs- ! ar e c oritinui^ hot the 

least of ' which is the time doctor s spend hourwise in reading/' 1 Some - 
; how, as Bill Ruhe s ays/ you can't get your hands on these things. Well, 
maybe that is because nobody tried hard enough in our former organiza- 
tioris'/ </The big danger, io me, 'C)f a formal organization is we don't 
recognize the nuts arid bbltsrof what goes on out in the practical world. 

f e r'e pr.ac tl'c a.1 ■- pr. obi e’ril s --■•y 6u 
meet in the world are that things aren't done the same in New York as 
/ ^hey arie ^inNe w Englanfro^ South^r ihe^W You - hay e to 

■ '’adapt td'^each area/- and 1 you can't give them a rigid set of standards: 

'■ : y. : . > ; ,V V- 
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I think Bill Ruhe is very aware of this. We don't want to lock 
the doctor in a single system. . 



DR. BROWN: I am still very concerned about what we have 
accredited. We are hurting others and giving a blessing to those kinds 
of learning experiences that we know from study after study are less 
likely to effect behavioral change in physicians iri improving patient 
care. I think it is extremely unfortunate. .j\y . . ; ! 



DR. HOWARD: If you can give feed-in to Bill Ruhe, I think he 
will appreciate it because it will get to your Advisory Committee, and 
we need more feed-in. 



DR. MICHAEL: I will tell you what I was nodding my head 
about. I have just come back from a so-called course at a non- 
university affiliated hospital. They had a so-called "postgraduate " 
course. By all standards it was a magnificent success. Eight hundred 
fifty people deducted it from their income tax. But when we went to 
the institution and tried to look into some of these things, to stimulate 
some thought on what continuing education was, we were hitting a 
stone wall. "This is not what you are here for. ; You are here to put 
an asterisk by this course that is listed in the Journal. " 



So far as I am concerned, we didn't accomplish a thing, 
it is a success. There were eight hundred fifty people . 



Sure 



v DR. WINSTON MILLER: The Utilization Committee should be 
a tool for continuing education. It is manipulated by the big stick not 
by the carrots. Now, you are beginning to pass out carrots. You 
really ought to pass out carrots for the kind of programs that have the 
greatest impetus for the performance of medicine. 



DR. EVANS: Y ou are still only going halfway . 



: DR . MICHAEL : Not even halfway. 






DR. EVANS: Because the real guts of continuing education is 
what a physician does eyery day in the hospital that he works in, and 
how he and his colleagues review what.they, all do together, how they 
run departmental meetings, how they run audits, how they interact ; 
with the Board and administration to maintain standards of care. : 
These things are the guts; of continuing education,. 



The courses are nice. They are like icing on a/cake. 
attract the flies, and they taste pretty, but they aren't much good if 
there isn't some cake underneath them, that is worth eating. ™ ~ ’ ~ 



This is 
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the one thing that we all pass by, and this brings one group into this 
which isn't even scheduled to talk here today -- the Joint Commission 
on Accreditation of Hospitals. It seems to me that they are involved 
more deeply, probably, in accrediting continuing education or could 
be more deeply involved in it than, virtually, any other group, because 
regardless of how many courses you run, you can run conferences 
until the cows c ome home, the place where this guy is really going to 
learn is where he earns his money, makes his bed and eats his food 
everyday, and this is in his community hospital. 

DR. STEARNS: What makes you think the AMA, this Council — • 
Bill Ruhe's accrediting people -- are not going to give credit for the 
kinds of thing you are talking about ? I rather suspect they will. When 
they talked to us, they were most interested in this kind of activity in 
community hospitals, and I think they may very well approve such 
kinds of programs; ;y 

DR. EVANS: I din 't say they wouldn't, but they haven't. 

DR. HOWARD: May I make one -final comment here? In ° 
setting up the standards to what we will give point credit for in con- 
tinuing education, I had to write up the so-called essentials or 
description of what we would give credit to, and I included all of the 
things Bob and Clem arid Winston are thinking of, because, to me, 
this is the nuts and bolts of continuing education. 1 ■ u . ‘ 

You do it on the job, with your own patients, in your own 
hospital, and, to me it is the heart of continuing education. I would 
like to see AMA give credit for the very things mentioned. : 

CHAIRMAN HALL: ; I think what you are saying is we need to 
define the universe of continuing education in particular, and the evalu- 
ation procedures now accepted as the defined universe can be extended. 

Dr. Elliott is here to represent the American Hospital Associa- 
tion's concepts of the status of education program evaluations. 

• '* J DR1' - FREDERICK N.' ELLIOTT : As hospitals are beginning to 
identify part of; their purpose as being discreetly and distinctly educa- 
tional, and since they are developing an area of justification which has 
to do with education considered distinct to some extent from their 5 ;: > 
primary service function, they have an obligation to be concerned with 
such topics as this. 

^ the- evaluation oif their own commitments to education 

is essential because it is parallel with' the auditing procedure that ri i 
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involves the expenditure of money or the utilization of other resources. 
Therefore, it is morally and operationally a part and parcel of the. 
hospital's educational program, and to that extent it is essential. 

i ••• However, I lyas very much pleased to hear Dr. Rosinski say 

last night -- talking about the universe of education -- that if there is 
any instituti on in our society today which is undergoing a transforma- 
tion, partly voluntary and partly involuntary, it is certainly the hospital. 
In evaluating any type of activity, I assume that means measuring the 
achievement against the goal, and since the goals of the hospital are 
changing and are not too clearly comprehended in a contemporary way, 
let alone as far as the future is concerned, evaluation. here may have 
another purpose. Rather than evaluation being contemporary with the 
effort, it may be that evaluation can be used as a yardstick to project 
goals and to indicate a change in our effort or a change in che desired 
product. ) ; . • 

Now, we have both situations existing at the present time in 
. hospitals. The hospital education effort is presumably directed towards 
helping the hospital to meet its immediate obligation for high quality 
service or medical -care; and .ostensibly, to enable it to /respond flexibly 
to changes in circumstances, to become the kind of institution which 
it is going to need to be in the future. / ■ :i, :• L 

Many of us feel that the hospital of the future will become a 
very different type of organization, representing a ; concentration of the 
community's health care resources, defining health in terms that are 
both social and psychological as well as physical, and that it will be 
concerned, with education, prophylaxis, diagnosis, treatment of acute 
conditions, convalescent, care and services directed towards rehabili^ 
tation. . ,v-\. .' <7 / ,[ i: .*•$£ rj .. / 

We have two problems One is the contemporary problem of 
merely delivering medical care. The present program of delivering 
'ft medical care is bound inevitably to financial considerations. Wants 
are being changed into demands in the economic sense. 

We are being confronted with the impossibility of achieving a 
maximum potential because financially it is impossible to do since we 
are going to create demands which will forever outstrip our resources 
in the treatment of the illness.- iThis means that we are going to have 
. to redirect ourselves to becoming health care oriented, prophylactic ally 
1 educatipnally oriented, and we ar^/going to have to remove the 

morbidity that results from the misuse of, drugs,; alcohol, automobiles 
• and money. ».V V * n" C t-isi:. 
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If those are the two things, we have two problems. Evaluation 
in the hospital can only be effective in determining whether or not what 
we are producing is helping us to do a better job. So, it is extremely 
pragmatic and extremely unscientific, and it consists in adducing the 
evidence of better patient ca;re -- whether it is a reduction in unfortunate 
incidents or more rapid recoveries or more accurate diagnosis or more 
effective treatment. , n •:••••:•;/ _ -:x. •. :* 

Now, we have some very great deficiencies which are apparent 
to us even there. We lack numbers of these people, and they lack 
competence. In those simple areas we have three components of the 
educational process that we have to consider^ ! ' - 

In the educational process we have information, intellectual 
content, we have a skill content. Finally we have the intellectual con - 
tent and the skill content coordinated with the underlying emotional 
constitution or attitude of the individual in producing judgments. 

Now, the value of a judgment is the extent to which it is appro- 
priate to the stimulus that creates it, and we know a lot now about the 
information we should give, and we know a certain amount about the 
skills we should impart. We know very little about what we can do ' 
about the defects in judgment which result from the multiplication of 
knowledge and skill by an imperfect personality. 

So, at the beginning of the educational process, we must know 
what the product is that we want -- whether we want/to turn out a pilot 
or a football player who is basically a killer or a man who is going to 
become a healer.’ We have an obligation to analyze this individual the 
best we can;- ‘His acceptance into medical school or into the healing 
society is a crucial step in which we may have committed ourselves to 
failure. When we make that acceptance, when we accept "A/s" or an 
IQ or a demonstration of a physical skill as primary evidence of the 
ability to exercise good judgment,' we are omitting what is nine -tenths 
of the proposition,' and that is what kind of a person is this person, 
what are his neuroses, and how will they motivate him; : . : 

s 7 / ; ; So, we need to study that; and we need to do it much better than 
we are presently doing it, & Now, ; how do we influence these things ?- 1 
t How s caii you chaiige> attitucie s ? J W e can talk about supplication 1 or exhor- 
; tation or rewards or punishment;, but we have got very little scientific 
evidence that gives us a predictable; result from changes in the environ- 
1 ^ment; or from othfer pressuresi' psychological or' social, on the individual 
" which change his behavior patternSi'O In other words; :; how does he use 
his knowledge and skill to respond to a situation appropriately. ( 
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Now, this appears to be true, because we are beset with a 
tremendous deficit of professional judgment in our hospitals, even in 
the narrow concept of medical care. 

As we see the hospital becoming a different type of an agency, 
we can go back and take one more look. It needs a new measure of 
intellectual input in the areas of social mechanics, social anthropology, 
psychology, the mechanisms of relationships. We are finding already 
that there is a serious intellectual deficit in the understanding of people 
who are trained in a vocation as to how they and their vocation relate 
to other people, and what their real function in society is. All of our 
medical students -- and certainly in my generation -- were completely 
deprived of this. We were sent out strictly from hunger. We didn't 
understand the hospital as an authoritarian or social institution. We 
didn't understand our place in society, and we didn't really understand 
what we were addressing ourselves to. Fortunately, we got by, and 
we achieved some personal satisfactions. The problem that threatens 
the hospital as an organic entity today is the fact that the people who 
should be the "cream" of our intellectual crop are babes in arms in 
understanding the society in which they function, their relationship to 
other individuals, and their relationship to the organization. So, there 
is where we need more input intellectually. We know we need more 
training and less education for some people. There is a great deficit 
of skills, including those of communication between the hospital and 
the healer and Vais patients. We need a reorientation towards health 
from medicine. 

So, if we are going to evaluate we must first of all determine 
what the standard of values is to be. Today it is contemporary medical 
practice. Tomorrow it is a social concept of health, and we must 
measure the products against those. Knowledge and skills are by far 
the lesser of three components. We must measure the third compo- 
nent -- which really underlies judgment — the emotional bias of the 
individual. We must seek ways in which we can change his behavior 
in a significant way. 

There is a lot that we need to learn from you, and there is a 
lot that we are looking for you to do, and except for being, concerned 
and grateful, we have no further competence . We have no competence 
except our need. 

DR. HOWARD: Have you just really said, then, that what we 
need more of in training health service individuals is what are now; 
called "behavioral" sciences ? That is, experience and knowledge and 
skill in those fields which formerly, strong family ties gave the young 
individual who wanted to be a doctor or nurse ? Family ties are much 
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weaker now with cars and television sets and schools now taking over 
the child while not admitting the in loco parentis theory. 

DR. ELLIOTT: I think there is need for an intellectual input 
in social and behavioral sciences for those going into the healing art. 

I think there needs to be individual application of the skills psychology 
and psychiatry and social science for the individual, to teach him to 
understand himself, so he can be motivated to modify his behavior. 

DR. STEARNS: I wonder if I am not hearing something else 
come through here • — the social concept of health versus the standard 
approach to medical care as we now know it. You said the knowledge 
and the skills may be the least important of three items. What you 
are really talking about is a better system of medical care in which 
some people who have the total socio-economic picture in mind, will 
be able to send patients to some highly trained, but le&b knowledge- 
able people. Is this the total system that yOu are talking about? 

‘ DR. ELLIOTT: l am, because our hospitals are increasingly 
suffering from a deficit of trained people, not necessarily the highly 
intellectual pursuits. As our number of personnel go up from one and 
one-half to now ne airly three per patient, there is actually less contact 
with the patient in both physical terms and psychological terms. More 
and more of our patients, including those in the expensive private 
room, are suffering from what almost amounts to sensory deprivation. 

DR. McGUINNESS: May I explain that? I was a patient in our 
local hospital on four occasions. I Yearned more about that hospital 
in those four admissions than I did in seven years sitting on the Board 
of Trustees. In a one -week hospitalization, I had twenty -four different 
nurses taking care of me. I kept a roster of all of these twenty-four 
girls, and I tried to remember the names of a couple of them. I think 
this is one of the kinds of things that Fred is talking about. I felt 
deprived of somebody in there whom I could identify with other than 
my "doc" whom I saw briefly in the morning and in the evening. It 
was quite a revealing experience. 

DR. FREIYMANN : Fred, as usual, you open up new vistas to 
me. ' In talking about investigating the basic personality, the emotional 
bias -- I think is the way you put it — of the student, you made me 
think of the last couple of articles by Dan Funkerstein, analyzing the 
students admitted to three schools — Harvard, Colorado and another 
one.' 1 That was one of the most dynamic documents I have ever read. 

If you recall, he breaks them down into the student psychiatrist, the 
' student scientist, and says there is no room left for a student physician 
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DR. ELLIOTT: And that is the bias of your selection. 

CHAIRMAN HALL: What you are saying is we ought to have 
our evaluation procedure moved up in time. . , . 

DR. ELLIOTT: Exactly. 

DR. FREYMANN: Somebody said you should go back to the 
elementary school. 

DR. ELLIOTT: You have to evaluate the raw material before 
you can even start to evaluate the product. 

DR. STEARNS: Is it possible to put an input into that raw 
material once you have got them in there. It seems to me that medical 
students and interns and residents need to be confronted by the kind of 
doctor who is acquainted with the problems you have: , not having the 
same nurse; having one who does know how to hold your hand; who does 
know how to shunt you to the proper kinds of .recreational and peace-of- 
mind kind of situations. It is virtually impossible to get students, 
interns and residents to accept a visiting physician whose job is to 
teach them what I think you have put in its proper place. I am very 
hesitant about pushing this too far, when you talk about the knowledge 
and skills kind of thing, because this man is not the "red-hot'' internist, 
the "red-hot" endocrinologist, the "red-hot" this or that. 

Now, if this is going to be meaningful, it has got to have its 
input in the medical school. The medical school must get the student 
to accept that kind of teaching as part of his education. Otherwise, 
they discard it. They just won't accept this kind of thing. 

DR. ELLIOTT: Well, Dr. Miller the other night said he thought 
that attitudes could be changed. He used the idea of example, and I am 
sure that has a beneficial effect on many people. Further, we know that 
systems of rewards and punishment do have a dynamic effect on any- 
body's attitude. Unfortunately, we are so unscientific in this area this 
time. We know it will have an effect, but we don't know what the effect 
v will be . Punishing of an individual -- sending him to jail.— may reform 
him, and sometimes does, but it will frequently do the opposite.. Per- 
plexed parents are saying, "I was too strict, " and they are talking to 
somebody who says, "l was too permissive with mine, " and they have 
both got the same kind of a problem. I think we are in the infancy of 
being scientific toward such emotional things. Actually, we have an 
emotional reaction to students which disqualifies us from functioning. 

We like our students, or we dislike them, and we don't even analyze 
that. . - U Y '''v - ■ v;'V : \." -v--.- 
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DR. MIXTER: I think one of my most exciting and memorable 
experiences since I have been working on the staff of the AMA was the 
accreditation visit to the new medical school in New Mexico. They 
are producing a breed of doctor that you would be astonished at. 

What they are turning out is boys arid girls who are extremely 
conscious of their social context, of the resources of the community, 
their obligation to the community. They are taught to be communicators. 

CHAIRMAN HALL: Where did you learn your social awareness? 

DR. MIXTER: I got it beaten in to me. 

CHAIRMAN HALL: 5 As well as your skills and practice of 
■medicine. : ■ : ■■■:;:■ $<■>. -^ov; 

DR. MIXTER: Just got beaten in to me. ' ; 

CHAIRMAN HALL: Where? v'H 

DR. MIXTER: We/ l,i family primarily. ./ 

.v.. CHAIRMAN HALL: You learned it in the home environment, 
and think it 'was built in 



DR. MIXTER: I don’t know. All I learned in medical school 
was a lot of language, really. 7> 

. ROSINSKI : You learned social awareness in spite of 

medical' school? >v >7 V/ 

V ; DR. MIXTER: Surely. ; •; .L j;< . ;■ -V! 

DR. ELLIOTTr vI have a feeling that the people who acquire 
sociak awareness acquire it elsewhere, rather than in the formal 
educational proce s s *?•'. V . . 

: : DR . MIXTER : I am sure it wa s f true of those of us who acquired 

DR. ELLIOTT: The responsibility of a hospital is to be a 
humane institution in which the effect of a total environment, as well 
as the educational process is to produce the product we want — the 
humane physician. It may be that later we will want to raise execu- 
tioners who will cut down on the population or something. / So, you 
can't really evaluate until you have a standard of values. 
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Our present standard of values indicates we need more physi- 
cians who are healers. 

, DR. MIXTER: It is certainly true the only way to evaluate a 
hospital is go be sick in it. , • 

DR. GRAIG E. BOOHER: Until the George Millers and the Ed 
Rosinskis in this world come up with a little more sensitive evaluating 
tool, I wonder whether maybe we are stuck with the horrible conclusion 
that there is probably worse general medicine being practiced by 
general surgeons than surgery being practiced by general practitioners. 

DR. WITTEN: I was very interested this morning in hearing 
Bob Moore make a statement about the lack of adverse effect upon 
surgical training programs by Titles 18 and 19. The American Hospital 
Association was one of the groups that warned those of us who were 
high on Title 18. You warned us that Medicare would destroy the 
training programs in hospitals. . <•. •, ?.-i 

DR. ELLIOTT: Yes. 

DR. WITTEN: What does the American Hospital Association 
think now? Do you agree that there has been little or no adverse 
effect upon the training programs by Titles 18 and 19 ? 

DR. ELLIOTT: Well, I can't answer that in any kind of quanti- 
tative terms. I don't know. I suspect there has been very little . 

Also, I don't think the AHA warned that it would mean a destruc- 
tion of the teaching program. I think it meant there would be a destruc- 
tion of the existing methodology, which is quite a different matter. I 
believe that the removal of the medical indigent, as a passive care 
' .case on which to practice dissection and so forth,; has been replaced 
by an activity in which teaching is conducted by people who have a 
commitment to .teaching, including practitioners, and this is a tremen- 
;; dous upgrading, because it is no longer purely an economic base . It 
is on the basis of commitment. The removal of the indigent, threatens 
the supply of bodies, so to speak, but that in a thoroughly discredited 
system. It will be replaced by using people with dignity, with reverence, 
and with enthusiasm for good treatment in teaching. I don't think it 
can do anything but good.; v i:«; i: 

DR. KING: You missed the opportunity to set up a balance and 
find out .; So, y our statements ' are really, without observational fact or 
support in either instance. 






r DR. ELLIOTT: That's right. ' Just philosophy on my part. 

I don't know. ! V; 

DR. EVANS: Is the AHA attempting to evaluate the role of the 
administrator as an influential individual in creating a patient-oriented 
care institution? In many of our institutions, this is a basic problem. 

DR. ELLIOTT: This is the most crucial organizational question 
through which hospitals' are now evolving. We have a triumverate. A 
Board of Trustees filled with altruistic things and ideas of fiduciary 
responsibility. The administrator, promoting development. And a 
profession fighting for what was relevant twenty years ago. Living 
together in a state of armed truce is not creating the kind of organiza- 
tion which is doing even what it should do today, let alone what it has 
got to do tomorrow. ■ \’p' s f •. 



determined way. It is going on as a result of a lot of strife and 
Brownian movement and negotiation but, hopefully, something that is 
organic and intelligent will come out of it. 

DR. ANGELIDES: Don't you think, Fred, this has already 
been determined?" The Darling case, ; 1 think, pins it to the Board of 



■•■h DR . E LLIOTT: The Darling case has awakened a lot of board s 
to the area of their responsibility, and it has given a lot of mileage to 
people who want to push medical staffs around a little bit. As a matter 
of fact/ in the Province of Ontario now, the new Hospitals Act puts the 
onus on the Chief of Staff of a hospital to displace an attending physician 
and throw him off the case if the Chief doesn't like the way the case is 



Now, if that doesn't burden the profession with something it is 



Now; bringing about an organic unity is the most crucial thing 
that is going on. 




Now, it is not going on in any particularly intelligent, pre- 



it out. I was amazed to read the State of Michigan licensure statute 
regarding licensure of hospitals, which is a most lucid and detailed 



Trustees, regardless 6f what mechanism is subsequently held to carry 




being handled 



not ready for yet ! Don't interpret the law as reality. The law operates 
in an oscillating fashion, usually behind custom; sometimes embarras s • 
ingly ahead, but never where ‘custom is;’ 
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Now, the Darling case gave great impetus to laying responsi- 
bility on the Boards, and it has been a healthy thing. But if it is 
accepted as dogma and quoted like the prohibition against the corporate 
practice of medicine -- which is becoming more irrelevant daily and 
will be as dead as the doctrine of charitable immunity in another ten 
years --if this is quoted as written and as though it is going to control 
reality, we are making a big mistake. 

DR. ANGELIDES: I was more amazed by what the State of 
Michigan has written into the law than the Darling Case: . T think it is a 
much more lucid document and more comparable with the times. 

DR. ELLIOTT: I think so. 

CHAIRMAN HALL: We are speaking of evaluation processes -- 
evaluation of the end product, which relates back to staff privileges in 
the hospital; evaluation of the product in process, the graduate students; 
and, as you suggest, evaluation of the individual before he starts the 
processing. Can we look to the hospitals to generate the funds to do this ? 

DR. ELLIOTT: To the extent that hospitals generate the funds 
to undertake educational programs, I think they will accept evaluation 
procedures as part and parcel of those programs, to be apportioned 
appropriately. ; ' v \> •, ' -;c:' > 

You are up against another question now. To what extent is 
the hospital -- which derives its income from services; to patients — 
entitled to tax that inc ome to subsidize another enterprise. Now, that 
other enterprise, to me, should justify itself on its own merits.; If it 
is in the national interest, we know where the money should come from. 

If you are as old-fashioned as I. am, you think the individual 
should be motivated to provide his own improvement. There is hardly 
any hope of going back to that, but the point is, it is now getting -to be 
almost a national scandal, because the hospitals that have good educa- 
tional prograriis now are taxing their patients from four dollars to nine 
dollars a day for the 'benefit of the community at large and the hospitals 
at large. It just doesn't make any sense, and the treatment in those 
hospitals is not that much better. v - • ‘ 

DR. PERKINS: The reality of today is that the Darling Case 
has one hell cf an impact. . • '■.,■••• 

; • DR. ELLIOTT: It sure has. '\y .- : : '"i ; ?,■ ■ 
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DR. PERKINS: And you cannot stand there and philosophize. 
There are hospitals and there are programs and there are staffs and 
there are administrators and there are lawyers, and they are paying 
attention to it today, and maybe you are right they will' only pay atten- 
tion to it for ten years, but what havoc they are going to raise in ten 
years. : ■’ 

DR. ELLIOTT: Nothing has happened in law in the last twenty- 
five years in the hospital field that has anything like the impact of the 
Darling Case. • . ; '• 



DR. PERKINS: It is a reality of the workaday world to come 
back to what we are talking about, what is our universe. This is a bit 
of reality, and I don't think we can dispense with it in philosophic 
rhetoric. ' 

DR. ELLIOTT: Your universe is the question of the responsi- 
bility of the hospital, corporately, for the competence of care. That 
is where it derives its mandate to be concerned with the evaluation of 
its educational product. So, it fits together in a positive way. 



;/• « DR. HUSTED: The moment we talk about evaluation, we imme- 
diately jump into the formal educational structures. Immediately one 
starts talking about program, one starts talking about courses, one 
starts talking about medical schools. 



• ••• Yet in no medical school system is there a course in attitudes 

or communications . The physician comes out of medical school with 
a certain set of attitudes that are developed somewhere. I submit they 
are learned. They are gathered from the experience that he has had 
throughout his training in medical school and internship and his resi- 
dency, and I would encourage — in fact, I would urge you when you 
are looking at evaluation, not only to concentrate on the tOrmal, but 
take a tight look at the informal interactions. I have a sneaking suspi- 
cion that it is at these levels where attitudes and communications 
: ^techniques are learned. And they are learned. They are not some- 
thing that is just absorbed. ; :• *'• 

DR. ELLIOTT: Thank you for saying what I was trying to say. • 



That puts it right on the line. 



DR. EVANS: I think we are now forced to go one step beyond 
the Darling to a New Bedford decision which says, in effect, that not 
only must the physician be competent to practice at the level in the 
community in which he is working, he must now be competent to prac- 
tice at the national level of competence. This puts the pressure for 
evaluation of output directly back onto the whole hospital organization. 
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DR. ELLIOTT: That is a change in the interpretation of 
negligence from what is locally acceptable to what is nationally , 
acceptable. 

DR. EVANS: That's right, and that, of course, obviates what 
has been the major defense of malpractice for fifty years. 

DR. ELLIOTT: It is no longer any good. 

DR. FREYMANN: That case hasn't been finally settled, has it? 

DR. ELLIOTT: It will be. Though as we get the dispersion of 
knowledge in an ever-increasing rate, courts are not going to protect 
pockets of ignorance. 

DR. ELLING: Would we know what the national level of com- 
petence was ? ; 

CHAIRMAN HALL: Our next speaker is Cecilia Conrath, 
Division of Regional Medical Programs, Department of Health, Educa- 
tion and Welfare, Washington, D. C. 

MISS CECILIA CONRATH: The Regional Medical Programs 
are now three years old. We are going into our fourth year. Many of 
the activities which are being supported under Regional Medical Pro- 
grams are in the area of continuing education. If we consider demon- 
strations of patient care included with continuing education activities, 
approximately seventy, per cent of all of the support of Regional Medical 
Programs would be in the area of continuing education and training. 

As many of you know, there are now twenty-seven of the fifty- 
five Regional Medical Programs that are operational. 

: vr , As far as the kinds of programs and as far as the kinds of 
activities in continuing education, they go all over the waterfront., 

As.-far as evaluation is concerned; we are. beginning to see a 
few breakthroughs, and i would like : f .o make reference at this point to 
an activity which the Division of Regional Medical Programs started, 
when Frank Husted was with us took.ca full force. Frank really ..... 
deserves the credit for the three contracts we have at the present . 
time. Ope with Dr. Miller, ^University of Illinois; one with Dr. Steve 
Abramson, University of Southern California, and a third with Ralph 
Inge r sol at Ohio State. These are programs which address themselves 
to the shortage of manpower in the field of educational evaluation. 

These are training programs at the graduate level which will, hopefully. 
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begin to make some inroads on attacking both the process of evaluation 
as well as the product of evaluation. ' 

i. ! 1 - 1 •• 1 ' ‘ )..i ’ 

In the Division of Regional Medical Programs and individual 
Regional Medical Programs, we are interested in projects as well as 
total program evaluation. The decisions that will be made about 
March of 1970 when we come up for the extension of our legislation 
hopefully will be made with objective data. Those decisions will 
largely be made on hope and faith in the future as well as the confi- 
dence that what has taken place in the last few years has indeed 
contributed to the health care system in this country. 

The mood of the country how is one of consolidation and diges- 
tion, according to the Washington Post, of the legislative program of 
the Kennedy- Johns on years. There are many hopeful signs for improve- 
ment of programming, because I think the mood now is definitely to look 
for accomplishments and importance, to look at what are the objectives. 
What do you really plan to accomplish with X amount of staff and X 
amount of money and X amount of materials ?• 

We had a meeting about two weeks ago on an ad hoc basis to 
look at how you begin to assess a national program, and I mean national 
program only in the sense of a feedback to Congress on Public Law 

89-239. ? ; 'JA: . y-: -or, -U-. ;.'>i 

■ ’ ‘If we were a national program, 1 it might be easier, but we are 

not a national program. We are fifty-five Regional Medical Programs, 
and decision-making at the regional level is, indeed, our strong point. 
We believe that we have to work with the fifty-five regions in order to 
begin to give the kind of feedback which is the intent of the legislation. 
The whole tenor of Congress is going to be one of much more conserva- 
tive mood. In this connection we have taken a look at what is happening 
outside of medicine, arid what is happening in the field of higher educa- 
tion in general, to begin to see if we can get some clues from what is 
happening in the field of adult education across the country. - ;V; 

There had recently been completed a survey of aciult education 
which will come out ab out May of this year. This is a repeat of the 
1962 study, when Francis Kepple first became Commissioner of Educa- 
tion and contracted with Boston University to do a survey for the first 
time across the federal establishment and across the voluntary sector 
to find out what is going on, and who is doing what; and where are the 
facilities, and what kind of manpower resources exist in the country 
to bring about a more meaningful educational system, responsive to 
adults in today's society. 
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I think that the recommendations of this particular survey 
which address themselves to what can be the federal government's 
policy in regard to continuing education, may have meaning for medi- 
cine as well as other fields. 

This and some of the foment that is going on in the field of 
higher education has resulted in the creation Directors of Continuing 
Education at the provost level of universities. I think that we are 
beginning to see some of the reward system, not, perhaps, in the 
health sciences, but in other fields. This may be one of the signs 
that we need to keep in touch with and begin to use some of Dr. Elliott's 
observations about being in tune with what is happening to society in 
general. 



DR. FREYMANN: Ceci, I am amazed how often recently 
agriculture gets brought into this. 



MISS CONRATH: Yes. The movement in Connecticut if prob- 
ably one of the best examples of this, where the general extension and 
the agriculture coop extension have merged, and they are now engaged 
in a most interesting "retooling" program of converting county agents — 
who formerly were addressing themselves to problems of the rural j 
population -- into urban agents. This innovation— how do you convert 
from one field of activity into another field of activity, what is this 
conversion process, and how do you change the attitude and philosophies - 
may be one which will give us some clues as to how to shortcut some of 
the problems that we have in training. 

DR. HUSTED: To what extent are the Regional Medical Pro- 
grams moving into medical care for the disadvantaged as a broadened 
area of activity? V"V "• ; 



MISS CONRATH: I think this is a hot problem in many regions. 
We are closer and closer in touch with the troubles of Regional Advisory 
Groups. I think it is at this level where programs for the disadvantaged 
either pass or don't pass. 



I think we are going to see much more cooperative effort with 
OEO's Neighborhood Health Center Program. There are a couple of 
places in the country where we do have joint sponsorship, but we have 
not seen as much of this as I am told is actually going on. Many of 
these programs have not been able to build the back-up that the Regional 



Advisory Group feels is necessary, so they are returned for revision, 
and we don't see them, but there is more going on. : 1 v 
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DR. HOWARD: Is Comprehensive Health Planning going to be 
a big health factor fairly soon in accomplishing this sort of goal? 

MISS CONRATH: It already is, I think. In many regions there 
is a heavy overlap on advisory group membership, on committee mem- 
bership, between Comprehensive Health Planning and Regional Advisory 
Groups. I think the hang-up in Comprehensive Health Planning has been 
the funding part. •• ; , .‘a -'*r/ i..-- --.vr. g; \ 

We see this as a compatible and not a collision c urse. This 
is not to say it is one without struggles, but it tends to be complemen- 
tary, and there are a few places in the country where this has worked 
out fairly well. V i . . . ... 

jg DR. HUSTED:, Is impossible for the same advisory group, say. 
Regional Advisory Group, an advisory group in a region, to be advisory 
group to both organizations, to RMP and to 749 ? 

MISS CONRATH: This has been tried with varying degrees of 
success,: ranging, from, failure, where it just didn't work out, - to places 
where there is heavy overlap. •. I am not sure there is a place where 
there is a hundred per cent. ^ •, 

DR. HUSTED: By ’'overlap 11 you mean the same people are 
serving on both ? v. ^p., ■■■■:! ! iu •; .. ; •, -.■.•vr.-i' 

MISS CONRATH: Same people are serving on both, and two 
separate, different missions. Some different membership, but a 
considerable amount of the same.;; ^ . 

CHAIRMAN HALL: The legislative guidelines almost required 
different kinds, so it couldn't be superimposed. 



„;.-v DR. HUSTED: 
wanted; to be sure . , 



A different makeup, yes. I: realize.; that, but I 



CHAIRMAN HALL: Ceci, you have mentioned. that Regional 
Medical Programs is providing an important resource -- trained 
people — to use in the evaluation process.;;; Will. this resource be 
restrained ? I presume it is just establishment of the educational 
resource that one: needs to begin this kind of expertise. They will, 
then, determine their own; course of action. V. V '.s >. . n ■ 






MISS CONRATH: .Right.., The people who complete the course 
select the positions which are most attractive to them. As it turns 
out, the majority have taken positions either with RMP or with RMP 









» responsibilities. The Division does not make any requirements at 

all. The individual makes his own choice. 

CHAIRMAN HALL: I still go back to my bias of the meeting 
here. I would hope that something will come from the urging of the 
specialty boards, the re -enforcement of the hospital settings and 
possibly with the hammer of the Joint Commission saying that these 
kinds of: evaluation procedures are important as the continuum of the 
evaluation of education. For the student, we have him evaluated before 
he comes into our hospital; once in our hospital in a graduate program, 
he has an evaluative procedure; and as he continues to practice in our. 
hospital, he undergoes evaluative procedure. 

If we can get that kind of motivation, I think that we are going 
to make great strides in this process that will be very important, and 
the resource people are going to be a key. We can't make strides 
without the resource people. 

To take us back a little bit more to the specifics of graduate 
education, we are going to now hear from the American Board of 
Obstetrics and Gynecology and Dr. David Danforth. 

DR. DANFORTH: The American Board of Obstetrics and 
Gynecology is one of the oldest boards, and I do feel that I must empha- 
size this. The Board started to be organized in 1927 and was officially 
set up in 1930. , : ; 

In recent years it has been composed of fifteen members. 

They have been three members each from the American College of 
Obstetrics and Gynecologists, from the American Gynecological 
Society, and from the American Association of Obstetricians and 
Gynecologists. In addition, there are three members at large, who 
are elected from former members of the Board. The members at 
large are elected each year . 7 The other members of the Board serve 
for periods of six years. ; ■ 

One of the major concerns of the Board is in granting a certi- 
ficate to individuals who have served in approved training programs 
i .ahdiwho have been examined and found in the opinion of the Board to 
; be qualif ied as a specialist in obstetrics and gynecology. r :, ' ; 

The Board requirement with regard to training is three years 
divided equally between obstetrics and gynecology. Some programs 
are of four-year duration, the notion being that the training of the 
individual can be strengthened in his particular hospital by adding one 
• year, which may be partly elective. It may include certain assigned 
programs, or it may be research. ; !-v. 

113 



1 



The mechanisms that the Board uses for arriving at a conclusion 
regarding the abilities of the candidates are, first, a written examina- 
tion, given at the conclusion of the residency period. Immediately the 
candidate completes his residency, ■ he has a written examination of 
three hours' duration of the kind already mentioned -- the multiple 
choice and so on. If he passes that, he is qualified to take the oral 
examination eighteen months after he has passed: the written. The 
oral examination is approximately one and one -half hours duration and 
is designed to assess not necessarily the facts that the candidate has 
amassed in the previous several years, but,, rather, to test his judg- 
ment and his ability to use these :f acts that he has learned. I think, 
to all boards, the idea of an oral examination is an attempt to appraise 
an individual's judgment and his ability to solve problems. 

Now, in addition to this particular kind of evaluation, rather 
recently our Board has addressed itself to the possibility of an in- 
training examination. • : * 

We have approached this rather slowly, and the objectives that 
we have had are essentially those that were outlined this morning by 
Dr. Gregory. We hope that this, first of all, will point out to the candi- 
date himself what his learning is, how he is moving along in comparison 
to other men in different programs, i'- We hope, also, that 'it may assist 
in evaluating she program. . ’ 

I am a little more uncertain of this, however, because the kind 
of individual that goes into these programs is not at all uniform. Some 
so-called poor programs will produce many excellent, extremely com- 
petent physicians, whereas some of the best programs will produce 
men of much less competence. Consequently, I am hot certain that 
this will be helpful in evaluating the program itself . Surely, it may 
assist us in finding areas of common fault in programs. That is, 
where certain subjects are repeatedly sluffed off as not being of impor- 
tance.. It could well be that this could alert us, at least, to the possi- 
bility of strengthening. v • 

We have done very little with the notion of evaluation of the man 
who has been specialized.:.; In our Board, we are extremely mindful of 
the, challenge of the Millis Report, indicating the demand by the public . 
for excellence in their physicians;:' and .the admonition that we proceed 
as promptly as possible to try to evaluate just what kind of medical 
care they are delivering. . hX . j . . 

v-rr ; 'We. have discussed, this at considerable lengths, ; and, unfor- 
tunately; despite Our best efforts , we have not been able to come up 
with any means of assessing the abilities of the group of practitioners 
on the. national level. ■-■XXh ... vSV?,- i ■ •. o- : r 
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Our conclusion up to now, at least, has been that such self- 
evaluation should be made, and there must be concerted effort to 
improve or surely to maintain the qualities of excellence in the various 
institutions, but this almost has got to be done, we have felt up to now, 
at least, on a local level. , 

From our vantage point, the two major needs at the, moment 
are the elevation of standards of the individual training; programs and 
the maintenance of the activities of the man who was certified as a 
specialist several years ago and may have begun to fall by the wayside,, 
We have discussed the. possibility of recertification, as Dr. Witten 
mentioned this morning. This is sort of in limbo. We haven't exactly 
rejected the idea, but we also haven't accepted it, and it is very diffi- 
cult. The Board has been well-established. To suddenly announce to 
the people who have been granted a certificate with the understanding 
that this is going to be a one-shot affair, that you are new going to be 
certified or failed, to find out suddenly that this certification is in 
jeopardy, and he must go through this again, is rather a difficult 
problem. It also is a question as to whether,, this rightly concerns the 
American Boards or whether this, again, is a problem that should be 
dealt with at the local level. 

DR. GEORGE MILLER: You questioned whether the oral 
examination can iest judgments. I would phrase the question some- 
what differently -- not whether it can, but whether it does. It is not 
the instrument alone that is important, but the way the instrument is 
used. • .... ; r .. .,••• ; . ,. r . 

As an illustration, the American Board of Orthopaedic Surgery 
used the oral examination for the purpose of assessing candidates' 
skill in making professional judgments and solving problems. As a 
part of our study of the examination process four years ago, we did a 
careful observational analysis of what took place between examiners 
and candidates in one hundred forty half-hour examinations from which 
some generalizations could be developed. What we found was that for 
the most part, it was simple interrogation with the simple calculation 
of the number of queries and responses divided into the total amount 
of time. There was a question and answer every forty-five seconds, 
which is about the rate at which a standard multiple choice test of 
information recall is utilized. T "f H- 3 • >' ! 

In the course of the next three years, those oral examinations 
, were transform ed to ; probe judgme nt and problem- s oiving, and in a 
subsequent systematic analysis, following training of examiners and 
the development of standardized material, focusing attention upon this 
as a particular goal, when the data were now accumulated, it was a 
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vastly different procedure. No longer was it an interrogation, but, 
really, an opportunity for a candidate to demonstrate his skill at 
making judgments about specific problems. 

Now, each is called an oral examination, but they are very 
different things. If we want examiners to do certain things, it seems 
obligatory to be sure that they are doing those things, and if they are 
not, then, help them to learn how to do them in order to be sure! they 
are done. ■•••■ v; 

DR. WITTEN: Old style oral examinations are no longer justi- 
fiable, and I believe that all of the boards will change in time. The 
ultimate measure of quality of any examination is determination of its 
reliability. Reliability, in this instance, refers to the degree to which 
the examination can be relied upon to provide consistent and reprodu- 
cible results. An examination is considered to be reliable if it can 
produce consistent scores for individuals or groups of individuals if 
they were to be tested on repeated occasions. It is this essential 
reliability characteristic that is so often lacking in essay or oral 
examinations, which depend too much upon subjective judgments. 

On this matter of recertification, we have been asked by several 
boards how we would propose that they might establish a recertification 
process. It seems to me, that, as you mentioned, when the certificate 
of a diplomate of any board has been issued, it has been issued on the 
basis of being irrevocable, and I think that the boards should and must 
hold to this. This is my opinion. I am not talking of a legal opinion, 
but I believe that they are morally bound to that. 

But you have two opportunities, it seems to me. One is to 
establish the recertification criteria for all those who enter the Board 
certification in the future, all future diplomates after such and such a 
date. Secondly, to make it possible that those who are currently diplo- 
mates to voluntarily be recertified during their time in the service to 
patients. ■ • • : .. ; • 

In this way, I think that any established board can bring about 
recertification of the diplomates. v: ‘ 

DR. GEORGE MILLER: I must challenge Dr. Witten's abrupt 
dismissal of the oral examination, because I think he has touched upon 
only one aspect;! that of reliability. The more important, even more 
critical aspect of assessment procedures is validity. Does it, in fact, 
assess what you want to know about somebody. '-V-. 
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There are a great many things we want to know about practi- 
tioners that cannot be tested by written examinations of a very high 
reliability. Our task is that of increasing the. reliability of those 
instruments that have a higher level of validity. 



DR. McGUINNESS: I want to ask Dr. Danforth why the eighteen 
month wait between the written and the oral. I have a reason for asking 
that, of course. To the foreign medical graduate that certification 
means a great deal when he gets back to his own country. In Asia, he 
can use that in lieu of Royal Society membership to hold certain posi- 
tions in his faculties and in the government and so on, and because of 
his limited duration of his stay in this country, many of them can't 
wait that eighteen months. What is the reason for the eighteen months 
aside from the fact that it has always been that way ? 

DR. DANFORTH: Well, it just happens, incidentally, that 
although this is not a motivating factor, that this foreign medical y . 
graduate who is going to take his certificate home and hang it on the 
wall and use it as entry to practically all of his local inner circles 
can't do this unless he is going to stay here. In other words, he does 
not qualify to take the oral examination unless he has been here for a 
year and one -half after this. , 

DR. McGUINNESS: Why, why? 

DR. DANFORTH: According to the Board, regulations. 

DR. McGUINNESS: Why does the Board have such a regulation? 

DR. DANFORTH: There are several reasons. My understanding 
of it is that during this period of time he is in practice, and he estab- 
lishes certain relationships with the people with whom he is working. 

He indicates his qualifications to practice obstetrics -gynecology and, 
also, he shows something of his ethical standards. 

During this year and one -half, he will have operated on a good 
many women, quite a sizable number. I think we will be able to know 
a little bit more about him than we would if we examined him immedi- 
ately. We will know, whether he is actually limiting his work to OB-Gyn 
what is his ability as a practitioner. 

DR. McGUINNESS: Again, I don't deny that, and this is not the 
forum to press my point except it gives us a great deal of problem 
when we on the one hand are anxious to have these people take back 
their evidence of training, and, secondly, we are anxious to have them 
leave the country as soon as possible. 




DR. DANFORTH: We do issue them a certificate if they are 
going to leave the country after they have passed the written examina- 
tion, the first examination. 

DR. McGUINNESS: What kind of certificate ? 

DR. DANFORTH: A certificate that states they have passed 
a written examination in obstetrics -gynecology that is printed by the 
American Board of Obstetrics and Gynecology. !'■' 

DR. McGUINNESS: That isn't going to do the guy a bit of g >od 
when he applies for his job in India or Pakistai. . 

DR. MICHAEL: Actually, perhaps, this is a good idea, this 
waiting period. It gets to one item that Dr. Elliott was talking about 
in his presentation, how that man interacts with his community, is he 
a person, or is he just a technician. 

The question I want to ask is one of evaluation. You made the 
statement in the rules of the Board exams, some men from good pro- 
grams did lousy, and some from lousy programs did" well. It's this 
old problem, how do we evaluate what is a good and what is a bad 
program? 

I know we have the figures of how many deliveries and how 
many operations they do, but how effective is evaluation of a program? 

DR. MIXTER: Well, it is pretty damn effective, as a matter 
of fact. The least good programs attract the least apt pupils, and they 
expire after a time. It becomes evident that they are not any good, 
and once a program has gotten into that state of affairs, it takes an 
almighty effort to boost it up. 7 ;' ; 

Now, the Review Committee has got a number of ways of judg- 
ing this. Basically, they accept what the Program Director says about 
himself and his program: : These are the statistics; these are the rota- 
tional schemes ; these are library fac ilities ; the _se are all of the nuts 
arid bolts V 7 '//■ 

Also, we have a field staff who go about confirming what these 
people say about themselves. Now, the field staff person may be a 
1 'pediatrician or obsietriciari^ gynecologist or an internist. We have all 
of those, but they evaluate programs in all specialties. 

; •>' ’ Now, in a tight spot -- and I won’t name the city — but Dr. 

Danforth undoubtedly recalls when a.n obstetrician-gynecologist was 
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sent to a city somewhat south and east of us and did a survey of four 
programs, all of which we sensed were in trouble. Following this the 
whole city got together, looked at its problems, and reconstituted one 
good program. 

Now, a great deal of the probationary approvals that the com- 
mittees hand out are based on relatively trivial matters such as a low 
autopsy ’rate, something like that. These are, more or less, excuses, 
but they do force the people in the program to look at themselves. 

The thing practically does work. 

DR. MICHAEL: You know, Studebak^r.was a good automobile. 
People didn't buy it, and it went out of business. I am not defending 
those that, you axed, not enough of them are axed, but I am sure that 
there are good ones that go out of business because not enough people 
buy them. - •/•' . 

CHAIRMAN HALL: I couldn't resist thinking that there are 
many very bad programs that are strongly supported by outstanding 
students . • : 

DR. DANFORTH: A couple of years ago under the aegis of the 
College, a so-called Council on Resident Education in Obstetrics' and 
Gynecology was formed. This Council is composed of six men from 
the College and six from specialty societies. It was formed for the 
purpose of looking into problems of residency education and has defined 
certain problems that need to be answered. One of the first was the 
need for some kind of in -training evaluation. 

In addition, soon to be set up are consultative committees who 
will appraise a given program that thinks it is in trouble or knows it 
is in trouble before the Council drops the bomb. 

In other words, these people will go out and will consult and 
will point up the weak spots and will indicate what they must do in 
order to improve. In addition, they plan to have a register of avail- 
able opportunities, y It will be kept current for those who wish to inquire 
about where they can go, and, also, how many places they have got to ' 
go. I think the Council on Residency Education, otherwise known as 
CREON, will perform important functions, not the least of them, I 
believe and hope, will be this in-training examination that the good 
Dr. Hubbard has been helping us with, and that will be done under the 
auspices of the National Board. : • 
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DR. GREGORY: Dr. McGuinness raised a point I would like 
to pursue him on. While Boards originally were contrived to certify 
to the competence of the practitioners of the various specialties with 
which they are concerned, they subsequently became concerned with 
the quality and the characteristics of the training programs that pro- 
duced the candidates they would certify. 

Now, in addition to that, they have been adopted without per- 
mission or consultation by a great many other societies. For example, 
practically every hospital staff society in this country utilized boards 
as their primary screening committee, for to gain membership in 
such a hospital staff society, you must (1) be Board eligible or (2) 
after a certain length of time. Board certified. 

Now, in addition to that, it Seems that the Royal Colleges of 
Europe and Asia.' and similar byways have decided that we shall be 
their screening committees also. We are acting in lieu of them, and 
this puts an onus on us, in a sense, that we simply can't put aside. 

We have the responsibility because we haven't done anything to stop 
this kind of thing. But a certain amount of chaos might ensue if, for 
any reason at all, boards were suddenly withdrawn as a part of the 
certifying bodies in this country. 

DR. ANGELIDES: I want to ask Dr. Danforth how many people 
are refused entry to the oral examination in Obstetrics and Gynecology 
or any specialty that is represented around this table, because of poor 
moral and ethical character? 

DR. DANFORTH: I couldn't answer that in terms of numbers. 
dr . McGuinness : we do. : ' 

DR. DANFORTH: There are several. Of course, it appears 
to be easier to be immoral in obstetrics and gynecology than it is in 
some other specialties; and when’these things do come to light, as 
they do/ this will be grounds for not admitting them to the examination. 

t i-' DR. STEARNS: Does this pertain to practitioners or people 
in their residency? ^ .• •■•••> : ■ '■ 

DR. ANGELIDES: This is the eighteen months after. 

? DR. MOORE: Every year our Board-turns down several candi- 
dates who have completed four or five years of residency, perhaps, in 
a well-known university program, because the Chief or others there 
bring out that this man has been immoral and whatnot. 
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CHAIRMAN HALL: I would like to now call on Alan Kaplan of 
the Bureau of Health Professions, Education and Manpower Training, 
Department of Health, Education, and Welfare. 

DR. ALAN KAPLAN: We are now under the new NIH adminis- 
tration, and are in the process of being reorganized internally. 

The Bureau has a deep feeling, as we all do, that as spenders 
of taxpayers' dollars, we are responsible for these dollars and should 
account for them in the best way possible. We also recognize the 
deficiencies in evaluation of continuing education and hope, by sponsor- 
ing conferences such as these, that we can bring about improvements. 

The Bureau is responsible for increasing the quantity and main- 
taining and improving the quality of health manpower in these United 
States as a national program*. The quantity, you can measure this in 
number. This year we had so many physicians, and next year so 
many, and you can say this is the increase, and you can say we spent 
X number of dollars, and this is what happened. You got an increase, 
but there are other things. 

Did we spend the dollars in the best way possible ? For instance, 
if we spent six hundred million dollars building six new medical schools 
and got six hundred new physicians graduating from there, was that a 
good expenditure for six hundred million dollars, or could we have 
spent six hundred million dollars by helping the existing medical 
schools to expand their classes each by six— or each by twelve -- 
for the same amount of money or for less money gotten more physicians 
out? These are some of the problems we are faced with in evaluation. 

When it gets to maintaining or improving the quality of medical 
manpower, health manpower, you know the state. I can't describe it 
to you because we are all in the same boat. How do we measure the 
quality ? So, we are very, very interested in what comes out of this 
and the recommendations that you make and the work that you are doing. 
Hopefully, as time develops and more people get interested in this prob- 
lem, we can come up with answers. 

CHAIRMAN HALL: Dr. Kaplan, is Manpower doing anything 
about the evaluation shortage other than demanding this in the project ? 

DR. KAPLAN: Evaluation shortage ? What do you mean by 
"shortage"? 

CHAIRMAN HALL: Well, there is one. There is a shortage 
of manpower: . There- is a shortage of tools and evaluation of all of this. 



DR. ANGELIDES: Is it in the numbers ? 

DR. KAPLAN: I think we are doing this in examining why 
there is a shortage. What we mean by a "shortage"? Do we actually 
have a shortage? First of all; what area are we talking about when 
we say there is a physician shortage ? After all, we know that many 
times the things the physician does in his office could be done by some- 
one else. There are many things that patients te 1 ! physicians, and if 
they knew how to tell physicians in a better way, they might be able to 
save physicians' time. There are many various aspects of this shortage 
problem.. It is not one strictly of numbers, and I think we are examin- 
ing this as a total picture before we start putting in a push. 

- CHAIRMAN HALL; Has the Bureau, then, addressed itself to 
definition of physicians ? Is it defining the universe of a physician and 
his 'ask?-’ 

DR. KAPLAN: I don't know whether the Bureau has defined it, 
as such. We have defined it. in physicians manpower as those people 
holding MD and DO degrees. Looking at the broad spectrum we take 
into account not only practicing physicians, but teachers, adminis- 
trators and researchers. 

DR. GREGORY: Dr. Kaplan, do you have any difficulty 
differentiating between shortage and maldistribution? 

DR. KAPLAN: I don't know really. I don't know whether we 
have defined "shortage" as yet, and whether we understand the mal- 
distribution problem as a matter of transport, or as a matter of the 
so-called "health care system. " It is still very nebulous and open for 
discussion. I don't think anybody has any hard answers yet. 

DR. MIXTER: Dr. Kaplan, then, you subscribe to the proposi- 
tion that the product of all of this is not just the doctor, it is health 
care. This whole' concept to me makes more sense than the numbers 

■game!. *•' Vi \ y- \ 

DR. KAPLAN: The prime aim of the Bureau is to improve 
patient ! card; y This is y.he prime aim. We are not in existence just to ■ 

■ put out. more physicians, but to improve patient care If patient care 
can be improved by increasing the number of doctors, that is one way 
of doirig it; ; " ! ^ t- A ■. _ 

DR. FREYMANN : Alan, I hope you would expand that. Patient 
means one who suffers. ; MDon't you think it would be better to have your 
goal as health care arid get a little preventive personnel in there ? /■ 









DR. KAPLAN: I think you take "patient" in its broadest sense 
as including preventive care. 

DR. FREYMANN: That is a typically negative doctor's point 
of view towards health as being the absence of disease. Don't fall into 
it. 

DR. KAPLAN: No, I was just going to say if we had good 
preventive care, there might not be the mal-distribution problem or 
shortage problems we talk about. If we can encourage patients to get 
care earlier and encourage physicians to do these simple things for 
preventive care, we might alleviate a lot of the problems. 

DR. FREYMANN: If you eliminate atmospheric pollution and 
cigarettes, we can eliminate one of the American Boards -- Thoracic 
Surgery. 

DR. STEARNS: What do you call a person who comes into 
your office for a vaccination -- a health care candidate or patient? 

DR. FREYMANN: I was thinking in more global terms than 
that. I recall George James' aphorism .-- the laborer with a seventh 
grade education who is putting fluoride into the city water supply is 
doing more than the entire American Dental Association for the nation's 
health. If we think of health in terms cf a numbers game -- so many 
people to do certain tasks we confine this to a negative point of view. 
Have you ever seen a medical student write at the end of a history, 

"The patient is healthy. "? He writes, "No significant disease. " : 

If we think in a broader term of health as a positive thing, 
then, we expand our concept of health personnel. 

DR. KAPLAN: But whose fault is it the patient only comes to 
them with no negative disease ? Shouldn't we educate the patient to 
come to us when they are well? 

CHAIRMAN HALL : What do we do with them ? We negate the 
educational practice. They come in, and we say, "Why are you taking 
our time ? " 

DR. KAPLAN: That is the part of the educational process we 
h ; ve to go through. , * • 

CHAIRMAN HALL: We can't just address this- to patients. 

The reason I asked you the question about definition of the universe of 
the physician is that if we accept this medical degree phenomenon then 
we continue in a self-perpetuating statu s quo . 
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We must get the physician to accept the fact that he has a 
responsibility for health care judgments that should have a high degree 
of probability of success. I think the physician should be the leader 
who encourages this plan to throw the fluoride in the water. That is 
his responsibility as surely as it is his responsibility to choose the 
appropriate agent to combat pneumococcal pneumonia. 



DR. KAPLAN: This is essentially the philosophy behind RMP. 
Ceci's continuing education plan is not devoted strictly to the continu- 
ing education of physicians, but the health profession as a team. In 
the Bureau of Health Manpower, we are looking at it from a different 
point of view. We are focusing on providing the methodology of con- 
tinuing education, not putting on programs as such. It. is a total of 
continuing education that they are looking at -- not only the physician, 
but the dentist and the allied health people all as one team. 



CHAIRMAN HALL: This function is dependent on role identifi* 
cation, and the doctor 's role identification is fixed in medical school, 
and, maybe, during his internship. 



[ MISS CONRATH: I think there is a philosophical definition we 
are talking about here as well as the bureaucratic definition. I think 
this is what Alan is responding to. There may be agreement philoso- 
phically, but not in terms of organization. 



DR. HUSTED: There is also the problem of judicial delegation. 
The physician, first of all, must be i ntimately knowledgeable of the 
many and varied allied health people who are at his disposal or could 
be at his disposal if he made more use of them. Then he must judicially 
delegate patient care and preventive medicine to these individuals, with 
the probabilities that his time for more physician-oriented activities 
would be increased. 



DR. HUBBARD: Dr. Kaplan, would you care to offer any pre- 
diction as to what you see concerning the formal licensure of other 
members of the health team? I am thinking of the assistant physician, 
assistant pediatrician and these additional categories of individuals who 
are now in training. ; : J.'V •? 



DR. KAPLAN: I don't think I am really qualified to comment, 
as' such. __ 



DR. HUBBARD: You don't see any nudging from the federal 
lent? °-l O-lC •" J?'- : 



government? 
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DR. KAPLAN: There probably is something going on. I don't 
know of it personally. After all, the federal government is made up 
of people who are interested in doing certain things, and each person 
has an individual opinion. I think sometimes their personal opinions 
get involved in what the federal government is doing. 

CHAIRMAN HALL: Dr. Tucker, would you care to comment 
on that? 

DR. NORMAN TUCKER: As you know, we are interested in 
new ideas for contracts which relate to continuing education, and I 
would like to pose that question to him for a few moments of discus- 
sion. Dr. Gregory's in-training examination seems to be quite 
successful, and it also is being employed by neurosurgery. Could 
this be a mechanism for continuing education or, at least, for self- 
evaluation of the practicing orthopaedic surgeon? Could you, say, on 
a staggered basis a year after you gave an exam send copies of this 
exam to your practicing orthopaedic surgeon as a means for him to 
evaluate his present knowledge? If he found he was weak in a point or 
two, he could use this as a continuing education mechanism. 

DR. GREGORY: Yes, the noting that the in-training examina- 
tion might be utilized as a learning instrument at the same time is 
another lesson which we learned from Dr. Miller on this basis for the 
last three years. We leave the examination booklet with the candidate 
after the examination is taken, and he and his teachers can then go 
over it all at will. Mind you, the question booklet remained unkeyed, 
and no one knew the right answer, but that doesn't matter because the 
question is mulled over and dealt with by the students and the teacher 
collectively. 

Well, this surrenders security at once so far as the questions 
are concerned. 

Beginning two years ago, this examination was offered to all 
members of the Academy which sponsored this. The first year some 
four hundred people wrote in and requested the examination. Incident- 
ally, they were charged a fee for this. Of that number, I think a little 
over two hundred completed the examination and returned it. The 
results were handled anonymously precisely as they were for the candi- 
dates themselves. 

This year again it was offered. Some eight hundred took the 
offer, and it is my understanding that something over four hundred of 
those have been returned. 
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That examination may not be the most appropriate one because 
the kinds of things are to some extent more esoteric than the average 
practitioner finds himself comfortable with. I think this may account 
for the relatively low rate of return of examinations, but the willing- 
ness of the average practicing orthopaedist is expressed to the extent 
that they have already availed themselves of this opportunity. 

DR. WITTEN: Jack, I would like to comment on this willingness 
of the practicing physician to participate in the self-evaluation examina- 
tion. We learned from our internist friend this morning, and we heard 
from orthopaedic surgery, and you heard Dr. Stearns tell about the 
experience of the voluntary examination by those who took the exam 
that was put on by the Ohio and Connecticut Academies of General 
Practice, two state chapters. I might add they got into this business 
because the American Academy of General Practice would not step into 
it. Unfortunately, we were tied up with our certification efforts, and 
we thought this might dilute them somewhat. 

Nevertheless, the first time around this examination was given 
to thirteen hundred individuals, as I remember. They paid thirty 
dollars each for the privilege to take the examination. At that end of 
this time, some eight hundred had an opportunity to critique the exam, 
the critique brought about some changes in the second one. 

The second examination was given, and this time twenty-three 
hundred general practitioners voluntarily took the examination and paid 
the thirty dollars. 

DR. FREYMANN: Was this just in two states? 

DR. WITTEN: Two states are sponsoring it, but it is put on 
nationwide throughout the country, and I think this is very significant. 
The exam is not a one-shot deal. It was in nine sections in which they 
could sit down and evaluate themselves and -go to resource books and 
so forth. 

This examination came out in a series of one every two weeks 
or one per month or one every three weeks, and you had to have your 
results back in by a deadline. This change was very effective in our 
case, and I would just like to pay tribute to them for putting on this 
examination. 

DR. STEARNS: This exam has many interesting features. It 
is interesting that the Academies in Ohio and Connecticut came to 
Boston to the Postgraduate Medical Institute to ask them to make the 
examination. It is also true that if there are problems in the 



126 




vrrFs* 






wm r<S“*r.^^r^.rr!> i ntT'^rrT:/rr- , TrT..' *i?"'! ,, ?:T r “ >*,*.t ; *'f -V vY/ 1 ^ d!' ’ .’. '*. ■ ’ :*■ :>' ' « ' ■' ‘ ■. 



examination we can make a change because it is given serially over 
a period of time. This year we are very responsive to suggestions 
from the field and are adding new kinds of questions. Also, there is 
opportunity to give out bonus kinds of information things. You keep 
in contact with those who are taking the test, so that it is an interesting 
involvement procedure. 

CHAIRMAN HALL: . Maybe this is what Dr. Miller tells us 
that they are addressing themselves to -- what the student wants to 
know in the examination, and some of the student needs. 
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DR. STEARNS: I would point out, again, that it isn't what the 
students think they need; because we set that ground rule from the very 
beginning. We would ask them what they thought they needed, and what 
they wanted, but we insisted that this examination be made by special- 
ists in their own fields of practice — pediatrists make the pediatrics 
examination, OB people make the OB exam. So, it isn't the question 
of filling what are felt needs. 

CHAIRMAN HALL: We have thrown out the challenge this 
morning that it would be important for each of us to write down the 
commonalities of the various programs of evaluation that were dis- 
cussed here. I would like to start around the table and see what notes 
you have scribbled down on what things seemed to stick in your minds 
as important commonalities in evaluation relating to graduate educa- 
tion, and we have to very honestly add continuing education. 

Our program was very hotly debated almost a year ago as we 
set up the format for this coordinating conference as to whether to 
address it to continuing or graduate education. We finally chose gradu- 
ate education. I think that the group here has moved from graduate to 
continuing education. But I don't think it inappropriate that we have 
addressed ourselves to both problems a little. 

Well, I would like to call on Dr. Freymann to lead off. 

DR. FREYMANNd I won't throw you if I say I would like to 
start out by discussing nursing education. I think most of us would 
agree that in nursing education we have seen what happens when a 
group — largely academicians — assume the right to determine the 
duration and general content of educational programs by establishing 
accreditation requirements and establishing a need to pass national 
tests. This was done with little if any reference to the function that 
the nurses are going to serve. 
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Aims, you have given your example of twenty-four nurses 
taking care of you in your days in the hospital. I think nursing has 
fallen into a trap where education has very little relationship to func- 
tion. Now, I purposely started talking about nursing because we would 
all be delighted to stand around and kick the carcass of nursing educa- 
tion, but I think we are guilty of just exactly the same thing. So let's 
call a spade a spade here. 

Dr. Brem conceded that the Advisory Board on Medical Special- 
ties determines which boards are going to be boards without any know- 
ledge or information about whether the boards are needed or whether 
the people who want to have the boards are needed. All of the existing 
boards, have assumed the right to determine the content of educational 
programs. Most of them still determine the duration of the educational 
programs. Here it is perfectly obvious that orthopaedics and neuro- 
surgery are trying to break out of this bind. I don't mean to be critical 
across the board. The College of Physicians has assumed the right to 
determine what should be in the self-assessment program. All of these 
groups are working without real knowledge as to what is needed at the 
opposite end, what the physician should really be doing. 

Someone talked about arrogance this morning, and I would 
submit that, perhaps, it is just a little bit arrogant for anyone in the 
certifying boards to say that he knows what the doctor ought to know, 
without some data to back him up? 

I am sorry to say this in Dr. Butt's absence, because I would 
like to say it with him present so he could defend himself, but I took 
the College of Physicians' self-assessment test. It was lots of fun. 

It was an exercise in the interesting patient syndrome. Seven hundred 
fascinating questions, but I don't think it really had very much rela- 
tionship to the application of an internist to the practice of medicine. 

It brings to mind a letter written by Professor Ferrie at Harvard, 
an MD and a Professor of Biochemistry. There was a letter in the New 
England Journal of Medicine this year in which he pointed out there are 
many people with very few common diseases, and there are few people 
who have very many rare diseases. This, I think, is what the ACP 
test was aimed: at. • • ; 

I would like to say that we are witnessing here happily the fall 
of one orthodoxy. Thanks to the leadership of orthopaedics and neuro- 
surgery, we are turning away from the orthodoxy of a time requirement, 
and the orthodoxy of a blind progression -- without any evaluation -- to 
create a specialist. 



I would like to caution the gentlemen who are the iconoclasts — 

I think that is what you said. Dr. Gregory --to realize what a heresy 
this is, and I don't think any of us really realize how far this may go. 
Because if you can say that you can create an orthopaedist in the length 
of time it takes him to acquire certain knowledge and skills, and not 
in a rigid length of time, doesn't this really extend to undergraduate 
medical education, too? Aren't we really shaking the roots of the medi- 
cal schools? 

Just bear that in mind, that if we are seeing the disappearance 
of one orthodoxy.. I think we have to guard against another one which 
is just as bad if not worse. That is the one I was leading off with, and 
that is those with the self-assumed right, the self-assumed power, to 
set accreditation, to say that they know the needs of society, the 
demands of society, without actually measuring them. 

DR. EVANS: Jeff started off talking about the nursing educa- 
tion. None of us as physicians any longer has any right to talk about 
that. We abdicated the responsibility for it a long time ago. I would 
like to carry his analogy one step further. Nursing rather proudly 
proclaims as a hallmark of all of its programs the evaluation system 
built into them. In fact, evaluation for those of you who have tried to 
work with interline programs is a religion rather than a process. 
Unfortunately, they have made, I think, the ultimate mistake of address- 
ing evaluation to the efficiency of the educational system and not to what 
needs to be learned in this system. I would hope that as we go forward 
in evaluation and graduate continuing education, we don't ourselves fall 
into this trap, because content and need must be, I think, the absolute 
criteria against which anything else is measured, and you have to begin 
to measure these first. 

Secondly, I think the very great progression which Jeff has 
mentioned from graduate to continuing education shows the really 
inseparable relationship. Continuing education is really a basis for 
graduate education in an institutional setting, and I think it shows even 
much more clearly the one thing which we have skirted around, but 
haven't really hit hard, and that is the impossibility of separating con- 
tinuing education from evaluation of patient care in the setting where it 
is being given by those who are members of the continuing education 
program. 

DR.. MICHAEL: One theme that seems to run through a lot of 
the presentations is that when evaluation is considered in a program 
or residency or whatnot, it is usually crisis -oriented. 

We say that the program in orthopaedic surgery was not crisis- 
oriented. Well, indeed, it was. 
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Dr. Gregory pointed out that it was engendered by the fact that 
too many people were flunking the Board. There was a crisis. Why? 
We look at residency programs. I think it is safe to say that with rare 
exceptions, the self-evaluation takes place only if a crisis comes to 
the institution or accreditation is threatened. I think all this points 
out is we are all for evaluation. It is a good thing. It is here to stay, 
but it must be built into objectives in any type of program that is being 
designed or those that we have now. We can get away from some of 
the frantic crisis evaluation I think we indulge in. 

Let me come back to Jeff's statement about the example, the 
rare disease syndrome. I have talked with several internists about 
this, and ones that I would call more intelligent internists. They like 
this, and I asked them this question, 'When you have a patient with 
pneumonia, it really doesn't make a bit of difference if you know which 
way the DMA molecule coils, whether it spirals in the right or to the 
left or takes loops or whatnot. The patient has pneumonia. You give 
them penicillin. If they don't get well, you begin to scratch your head. " 




I think a lot of people -- and let's be arrogant again — internists, 
in particular, are interested in some of the esoteric things. These are 
the fun things of learning. 

DR. FREYMANN: I said it was fun. I questioned the value. 

Is it relevant? 1 

DR. MICHAEL: Yes, it is valuable because it keeps your 
computer oiled. 

DR. GREGORY: I think what I have heard here is some good 
news and some bad news. I have heard that certain of us are involved 
in efforts to break through, and that some of these efforts hold promise. 
I have also heard that there are areas that need to be identified, meas- 
ured; that don't have any prospect of immediate measurement or 
accurate identification. 

DR. WINSTON MILLER: I have been stimulated last night and 
today by what I think is an exciting threshold of progress in medicine. 
Certainly, it is highly encouraging to see we are ready to face mean- 
ingful evaluation methods of what we are, what we know, how we judge 
things and what we do with what we know. The humility of this is worth 
a great deal. •••' r:; ■ ■ J : y- •• : "• ' 

I also took the American College of Physicians test and share 
many of Jeff's views, but one of the chief values of the test was the 
humility it produced in most of the internists that took it. Even the 
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sub-specialists -- limiting themselves to one of the nine sub-specialty 
areas -- seldom were able to make 100 on the test. 

I think there is another highly significant thing that we have 
touched on today, and that is the shift of subject matter between the 
area of education and another. Basic science -- some of it -- is being 
shifted into the more advanced state from the pre -clinical years into 
the actual graduate educational program. This is going to produce an 
entirely different kind of a load on the graduate program, and we are 
beginning to see an increasing shift of clinical subject material from 
graduate programs into postgraduate or continuing education responsi- 
bility. 



The Society of Rheumatology has recently come out with this. 

It simply can't be taught in medical school. It has to be taught ip 
either graduate or continuing education programs. Psychiatry has 
virtually said the same thing. Narcology has said the same thing. 

I am sure that when we know about what we have to do we will find 
more and more of these kinds of things. The only way the services 
can be delivered to the public is to develop a greater bridge between 
the academic institution and the peripheral practice. I think we also 
see a fantastic change in concept with the development of the American 
Board of Family Practice. We are facing the issue of having all chiefs 
and no Indians, and we have previously relegated all efforts of insuffi- 
cient services to that nebulous group of Indians that was the last one 
present. Now that we are not going to have general practitioners to 
deliver the babies that the obstetricians don't want to deliver, we are 
going to have to look at whether we want obstetricians to deliver babies, 
or midwives. So, I think this was a stimulating conference and it is 
a shame that we can't extend this more to the practicing profession. 

DR. KAPLAN: Well, it seems to me that we are going through 
a challenging time. Everybody is challenging everything, whether it 
be the riots in the streets or somewhere else. The thing that I find 
very gratifying is that while we are challenging, we are not just tear- 
ing down. We are not challenging for the sake of challenge, but we 
are doing it with the idea that maybe we can make it better. I think 
this evaluation conference is a very important part of this, and I am 
glad to see it is going on. 

DR. WITTEN: I, too, found this Conference quite stimulating. 

I do hope that the product of this meeting may be distributed widely. 

I would have wished that each member of the Council on Medical Edu- 
cation of the American Medical Association might have participated. 

I wouldn't have been surprised if they might have learned a Tittle bit. 
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DR. BROWN: I guess I am, frankly, a little surprised at all 
of the activity that I have heard about in the area of evaluation, but I 
am a little concerned. I think about Carl White's statement that one 
should not confuse activity with action and both with progress. I am 
not sure I know how much progress has been made in the area of eval- 
uation because it seems to me what we are usually talking about is 
what I might call "second and third level" evaluation. The second 
level is evaluation and knowledge that someone has based on adminis- 
tration of some special instrument. The third level evaluation is even 
more often done, the evaluation of programs and not of the learner. 

The thing I think we have really got to get to is evaluation of practice 
and to see if there is any correlation between what we are usually 
evaluating now at the second and third levels and what physicians 
really do in their management of patients. 

I think this is where we need to be, and, finally, with respect 
to what has happened today or the effect of the evaluation of what has 
happened today, I think we might all ask ourselves will we do any one 
thing different as a result of having been here yesterday and today when 
we go back to whatever it is we go back to. The answer to that question 
will let us know whether or not this conference has been successful. 

DR. GEORGE MILLER: The title of this meeting was Consi- 
deration of Educational Programs Evaluation. I have been pondering 
the focal word which was "evaluation, " but the key word was "educa- 
tion. " Presumably it is about education that we are talking. I think 
of Paul Sanazaro's chart on education which has an input student body 
that comes into a black box where something happens, and, then, 
there is an output product. Our discussion is focused upon the input, 
how we identify those students who will be best. This kind of evalua- 
tion implies that if only they had what we wanted them to take away, 
we wouldn't have to do very much in the black box. 

Looking at the black box itself, it has two components, one of 
which is program organization, the setting in which learning can occur. 
To this we have given a good deal of attention. How do you evaluate 
programs? We have talked, I think often rather loosely about good 
and poor programs without establishing the criteria upon which these 
judgments are based. But we have looked very little at the educational 
process that is supposed to take place within that setting. To what 
extent are the principles of adult learning incorporated in this program? 
Systematic; observational analysis of the educational process is some- 
thing, to which we give almost no attention either at the undergraduate, 
the graduate or the continuing education level. Here again, I think we 
are inclined to use words rather loosely. We have focused attention 
primarily upon the product, the output, what the learner has at the 






conclusion of whatever the learning experience may be. Yet, there 
are those who have criticized this because it did not seem related to 
the real thing, the real life patient care. 

I think I would call attention to the fact that the product that is 
the objective of the education may be at three levels, immediate, 
intermediate and ultimate. When we are talking about patient care, 
we are talking about the ultimate product which is the critical and 
essential, but, also, the most difficult thing to get at. 

It might be profitable for us to direct some more of our atten- 
tion to the purpose of a four-week experience on internal medicine in 
a rotating internship. To what extent this purpose is achieved? To 
what extent does a continuing education program designed to increase 
a specific skill, increase that skill? 

The intermediate outcome would be the assessment of whether 
somebody takes it beyond that stage back into the setting in which he 
might use it. The ultimate, of course, is whether it does any good, 
but each of these is worthy of thoughtful attention. 

Finally, it seems to me that we have given little attention in 
the course of this discussion to more than talk about evaluation. Per- 
haps the next time, instead of exchanging information, it might be well 
to devote an even longer period to gaining some skill in the tools, the 
methods, the concepts, the application of evaluation so that we go 
beyond the level of intellectual assent to practice the better procedures 
of evaluation in all of the settings to which we devote our attention. 

DR. HUBBARD: I suspect that it is probably true that each of 
us came into the conference with his own, probably, pretty well-defined 
definition of what he meant by "evaluation, " evaluation of the individual 
or evaluation of the program, and to what extent we will go away from 
this conference with different ideas as to evaluation, I would not pre- 
sume to predict at this point. X think that I have seen some confusion, 
confusion of the objectives of the program with regard to evaluation or 
with regard to education and the learning process. 

Perhaps my view of evaluation is too sharply defined, too 
limited. I think in terms of the precise measurement of individuals 
which may, in turn, reflect the effectiveness of the learning experience 
or the training program that they have experienced. This came clearly 
into focus when Dr. Hugh Butt and; his committee first came to the 
National Board. What they said they wanted at that time was an evalu- 
ation of the effectiveness of the postgraduate training programs of the 
College, and to this end, they would set up an examination of the 



membership of the College on a voluntary basis which would reflect 
the effectiveness of the training program. As you have heard, they 
were advised not to go down the line of an examination, but, rather, 
a self-assessment program, from which no data would be derived. 

The objective changed. The same thing now is true of the other asso- 
ciations that are coming to us for similar activity -- even in the very 
word "self-assessment, " itself. 

I couldn't help but notice that this was what you referred to a 
moment ago, Carroll. I think this is a misnomer. Are we really 
concerned with the test? If so, we have got to recognize the fact that 
you have to sacrifice some values of the training program if what you 
want is an examination that is going to yield hard data. This, frankly, 
is what we like to see, an examination that can serve for evaluation 
of the individual or program or evaluation of the product or evaluation 
of the process. Then, I think, you have got to talk in terms of an 
examination under supervised control, and this is something very 
different from a self -learning kind of procedure. 

Just for a moment, I couldn't help being hooked by the bait, 
that Dr. Freymann tossed out a moment ago when he talked of the 
arrogance of the certifying boards, and asking if . . . 

DR. FREYMANN: I put a question mark after that. 

DR. HUBBARD: And asking the question whether certifying 
boards have the wisdom to know what a doctor should know. I appre- 
ciate the question. I have heard it before, but I would respond with a 
question. Who better should make this assessment? With respect to 
our own examinations of the National Board, we rely upon those who 
are expert in the field of surgery, pediatrics, biochemistry, pathology, 
the various subjects in which we examine and rely upon them to deter- 
mine what students should know. At the specialty board level, again, 

I asked the question, how better can we set up a system to determine 
what pediatricians should know than to ask those who are pediatricians 
to set the examination procedure. 

DR. FREYMANN: Look what the pediatricians themselves have 
done in ten years as they have looked to see what they have done. The 
second thing is who knows best if my definition of a core curriculum 
is everything I teach and half of what you teach. 

DR. BROWN: One source of objectives for an educational 
program is the experts in a: particular area. I think there are two 
others that are extremely important. (One is the learners themselves, 
and the other is society and its needs and wants. So, I think that. 
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maybe, specialty boards are completely irrelevant to both of those 
sources of educational objectives. Maybe, they are not always best. 

CHAIRMAN HALL: I think I would like to introduce one other 
thing besides the commonality now. I would like to see some recom- 
mendations, and I think Dr. Hubbard has given us this charge a little 
bit in his comment, that we should come up with some recommenda- 
tions. 

DR. HUBBARD: I opened one last evening. It wasn't received 
with much enthusiasm, but since you ask, I will state it again and that 
is something should be done to try to determine the competence of the 
foreign physicians who are in hospitals throughout the United States 
today. We know a lot about competence of graduates of American 
medical schools. We know very little about the competence of gradu- 
ates of foreign medical schools, with all credit to the ECFMG, and I 
have the feeling that if an organization such as this does not do some- 
thing about it, somebody else is going to clamor for an answer, and 
we may be pressed into a situation that we are not too enthusiastic 
about. 

DR. HEBERT: My most poignant reaction to last night and 
today is one of being really overwhelmed by the amount of people, 
agencies, boards that are involved in evaluation.- •< 

Now, please don't misunderstand me. I am not against evalua- 
tion, but my reaction was do so many agencies, boards and people 
really need to be involved in it, and could it not be more concentrated. 

It reminds .me of a thought I have often had that if a well-accredited, 
competent medical school graduates a physician, should not that diploma 
be sufficient to practice ? If a well-accredited and inspected training 
program intimately knows its own graduates, should: not that training 
program be able to issue his specialty certificate? My recommenda- 
tions are that we would try to consolidate our evaluative procedure. 

How, I don't know. >\ . 

f. DR. STEARNS: I was a little bit heartened last night to hear 

somebody remark that doctors, in fact, had been looking at what they 
had done for many, many years. I think it is easy to forget that many, 
if not most, physicians really try to be self -critical, that we were 
brought up on the case discussion kind of critical review of what we 
did, that we have been confronted with a lot of real kinds of almost 
harassment of how bad we are, and that kept coming out today. So, 
it was nice to hear somebody say that, maybe, we weren't all bad as 
doctor s.;? I don't really think we are. 



It is interesting, Fred, that there are a lot of people trying to 
do something in this field of evaluation, and I don't think there are too 
many. I think the job is terribly, terribly difficult, and I think one of 
the reasons it is terribly, terribly difficult is that we really are not 
all trying to do the same things, and we really don't have the tools that 
everyone will accept as applicable to every job. So we have to be pre- 
pared for a hard kind of undertaking which does need support. I would 
second George Miller's recommendation that, maybe, some programs 
and action, again, because we have tried it before, is what is needed 
at the present time. 

I wonder if it will be more productive, however. Namely, I 
am not sure that if we sat down and were doing evaluation things, it 
would be more productive, because if there is the magic of the tool 
here and the technique that can get across to us, it must have come 
by some other than the written word or the stated word. 

So, you know, maybe, a participated phenomena, and I would 
recommend that we try it. 

DR. McGUINNESS: 1 would imagine the future conferences 
would be -productive in proportion to advancement of the art of evalua- 
tion, and if the art docs not progress, then, the conferences aren't 
going to be much more productive. 

To go back briefly to last night, frankly, I had a little difficulty 
during Dr; Elling's presentation. Someone talked about those of us 
who got out of medical school twenty or thirty years ago. I got out 
thirty-eight years ago. We were talking a little bit about the problem 
of the whole patient and his environment/- 1 think we got that, a great 
deal of it, in medical school, because of the fact that a substantial 
portion of our instruction came from people who were in the practice 
of medicine. That was their primary interest, as opposed to today's 
full time faculties, the majority of whom are not in medical practice, 
and who are not too concerned with patients other than the diseases 
they have and their research problems. That, perhaps, is stating it 
a bit unfairly, but I am doing this to make a point. This is obviously 
something that has come about for two reasons ( 1 ) the explosion in 
science, and (2) the disproportionate research, disproportionate in 
the amount of funds available for research and teaching. If the govern- 
ment just produced funds for teaching in proportion to funds that have 
been made available for research, I think the picture would have been 
somewhat different .^;I think now we are getting more funds for teach- 
ings ■ arid, perhaps,; we can get more clinical faculty back in the schools . 
We will only be able to do that to the extent that there are; funds avail- 
able to support clinical faculties as opposed to research faculties. 
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This may help balance the situation, but I think we now are going to 
need the new kind of faculty member like Dr. Elling. 

i 

I can only say that -- as I said at the opening — what we are 
beginning to develop is expertise, Dr. Hubbard and his associate and 
Dr. Miller — we will progress to the extent that they and others like 
them probe deeper into the techniques of evaluation. 

ME. NORMAN TUCKER: As a non-medical observer, I think 
there is one commonality that comes through loud and clear to me, 
that somewhere in the ominous background, the greatest stimulous of 
all toward the evaluation of graduate and, more specifically, post- 
graduate medical education is this business of re -examination and/or 
ultimately recertification. 

DR. ANGELIDES: Certain things were discussed throughout 
this Conference which are difficult of accomplishment. At least, they 
have been stated, although they may not have been accepted. I think -- 
if you want to call this a recommendation — that education in the medi- 
cal setting, at least, should be related to patient care, and that 
education is a continuum that must be evaluated in its totality. Cur- 
rently the best we can do is study it in its traditional integral parts, 
and try to figure out how the interrelationships can take place for the 
future . 

I think, also, another thing emerges — that we continue to 
study, as Dr. Elling proposes, the universe. This is way off. Hope- 
fully, we can establish guidelines to help us in our day-by-day delibera- 
tions in trying to cause change and affect not only the traditional parts 
(graduate, continuing and undergraduate education), ■ tiut even sub-sets 
within each of these components. 

I think there is keen appreciation that there is need for skills 
outside of the traditional medical ones. I think that it is refreshing to 
know that people like Dr s. Rosinski and Elling have such a grasp of 
our problems. There is need for further experimentation since we 
know little about controls and the scientific approaches in these areas. 

The final thing we should consider as a recommendation is the 
attempt, wherever possible, to try to bring the component parts and 
stimulate interest between the disciplines to elucidate what is common 
and what is specific. This is the role of the American Board of Medi- 
cal Specialties. If this approach leads to a Commission on Medical 
Education, so be it. Regardless of present developments some 
mechanism must be organized so that there is. free interchange and 
interplay to evolve a more integrated system.: , : r iV . ;v - ,. r: 
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DR. HUSTED: It seems to me there are several elements of 
commonality that have come out that we ought to recognize. One of 
these is that evaluation process does have a function to serve within 
the educational sequence, and, also, that this has to be carried on 
with some functional frame of reference. 



Also, a second component is that of restraint. As we engage 
in evaluation process, not only must we consider the restraints that 
are imposed upon us by those who are being evaluated by the system, 
by structure, but, also, we must impose some restraints on ourselves 
and as we recognize the results of evaluation, indeed, impose other 
restraints. " \ ' : "'' f [ 

The third component would be one of utilization, and I would 
like to look at this in two frames of reference. One is the utilization 
of the evaluation product just to evaluate, and, then, not to do any- 
thing with the ultimate results. Even if they aren't particularly right, 
or even if they aren't completely valid, we still should do something 
with it. ' ' : ’ ■ ■ • t ' / ’ ■/ Y‘ ' 

The second aspect of utilization has to do with taking a look 
around your community for resource people who can be of assistance 
to you, and, indeed, who may be interested, and whose interests 
might be stimulated simply by contact. Take a look at the people in 
the schools of education, sociology, and testing and measurement 
fields. They may well be interested in working with you on specific 
problems of evaluation. ■ ' ' ' 

Obviously, a fourth item in evaluation process that has. some 
commonality is the student and where he fits into the picture not only 
in terms of behavior objectives, but in terms of his own needs, wants 
and desires, and where he perceives himself as going. 

The element of time is critical. I will refer particularly to 
the physician in the academic setting. You just don't have time to do 
all of the things that you expected to do. Yet, here for two days or a 
day and one -half , we have been talking about another kind of thing that 
ought to take up this time. . -* J-'.vi - V . ? - . y . 

’ ■ V ' You have to look at priorities,' what will evaluation do for you, 
arid when it has done this in terms of perception, is it sufficiently 
valuable and where does it ratejin your priority scale. Are there , 
other things that you might well have spent your time on. ; 1 ! r ‘ 

V;, That may sound a little heretical, but I still think it is some- 
thing you have to take a look at -- the relevance of the evaluation process. 
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We talked about tools, and I would like to talk about tailoring. 

You can't lift -- I don't care how good a tool that George might develop, 
or I might develop, or might be developed by one board or another 
board -- you just can't lift that tool and use it without doing some tailor- 
ing of that media or of that tool to fit your particular needs and peculiar 
situation. 

I also think that as you look at your evaluation process, you 
ought to be more innovative, or we ought to be more innovative. There 
are some electronic media, technology, TV and audio-visual terms we 
can use for analysis of whatever it is that we do in the evaluation pro- 
cess. I don't want you to forfeit objectives as commonality that runs 
through the whole of the evaluation process. Lastly, I would like you 
to remember, also, that there are the needs of the students, the needs 
of society and the needs of the academic process. 

I would caution you against frustration. It is very easy to 
become frustrated by the lack of progress we make in the evaluation 
sequence, and it is probably one of the most damaging attitudes that 
develop because of the state of the art or because we are impatient to 
get results. .. 

The recommendation I would like to see is more meetings like 
this. I would like to add to George Miller's comments my own educa- 
tional philosophy, which says you don't learn a damn thing until you 
are involved. I would like to see a conference where we get involved 
on a problem-solving sequence. I would like to work with some of the 
problems of the boards that have been mentioned here, not as an indi- 
vidual, but in a group and work through a whole evaluation sequence 
from the objectives criteria and measuring devices and implementation. 



You do it three times, and the fourth and fifth and sixth times 
become easier because structure of the process of evaluation, the 
sequence, becomes more workable. 



I would like to see an intensive workshop. I mean a workshop 
where we take our jackets off and get right down to the nitty-gritty of 
doing and get involved. :• ,, v «V* '• r ' u.:\- 



DR. PERKINS: In evaluating our techniques as teachers, we 
all can be reassured as to how much data and identification and feeling 
canbe imparted; within a tenrminute . period. 

DR. MIXTER: Well, it has got to be that there is one optimist 
in the crowd. I, think that I have learned something about evaluation, 
and that is that at whatever level it occurs, there are certain threads 
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that run through it, and there are two kinds of evaluation. One is the 
evaluation of the product that comes Out of the black box, and the other 
is, as Paul Sanazaro phrases it, "the black box itself. " 

As far as the product is concerned, it is apparent that one 
must evaluate the amount of language that has been acquired, the 
amount of knowledge that has been acquired, and the amount of wisdom 
and dexterity that have been acquired, the ability to cope with the spe- 
cific thing, and, finally, the ethos of the product. 

Now, this last one is the one we haven't got a handle on, and 
I think that is where our big hangup is, and it has been pointed out at 
the meeting, and I still think it is there. 

As far as the "black box" is concerned, I think there are three 
major components. There are the nuts and bolts, which are. easy. 
There is the organization, which is fairly easy, and there is the per- 
sonnel factor of the people who do the teaching. I think, there again, 
we haven't got the tools, and I think once one can see the problem in 
an orderly fashion, we can devise tools whereby these things can be 
evaluated, and I think we ought to keep on doing the things that we are 
doing and try to supplement them in the areas we know we are weak. 

I think this kind of a meeting is the place where thought is 
stimulated, and we go home and think about what Jeff said or what 
anybody else said, and, maybe, one of us will have a flash of insight. 

DR. DANFORTH: As the discussion unfolded, it seemed to 
me that evaluation was surely not an end in itself, but, rather, a 
means to some kind of end. 



• J I think our discussions here helped to clarify not only the evalu- 

ation process, but what kind of an end we are, perhaps, shooting for. 

I think', also, that the mere fact that we have spent this con- 
siderable amount of time and thought on evaluation in itself suggests 
that, perhaps, we are a little insecure . I think this is to the good, 
the insecurity, and questions that have been raised about are we doing 
this right, can we do it better, how can we appraise what we are up 

to, 

V.h's. 



I think; probably, is the first step to improving. 
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The question, then, arises, wtiat are we really trying to im- 
prove, and, quite obviously, the ultimate goal has been voiced quite 
^clearly this afternoon. This is delivering effective health care. This 
islwhat we' are all shooting forV ‘ I think all of these evaluation pro-* 

.. cesses that, we labor through are de s igne d to thi s end.' ; 
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I think it also is of great importance -- it surely is to me 
personally -- to have been able to.hear the discussions by the people 
in so many different areas. There are no two of us, probably, whose 
backgrounds or daily work are similar. Consequently, we are getting 
many viewpoints. I think it has been very important in an. effort to 
define these problems and to deal with them. 

As far as recommendations are concerned, I have none except 
that your suggestion. Jack, about having regional meetings. I think 
the more discussions of this kind there can be, exchange of views by 
knowledgeable people, the faster we will arrive at the place we want 
to get. 

DR. HOWARD: I think everyone here has tried to be practical. 
They have tried to adhere to the good principles which I learned in 
San Antonio or "the Chicago Loop" where objectives, method evalua- 
tion, and evaluation give rise to more need for more objectives. Now. 
everyone here said that we should involve the students or the pupils. 

I go along with that, and I think we should involve the. recipients of the 
health care somewhat more strongly than we have and listen to their 
voice. I think the biggest thing we can do in the future, as a recom- 
mendation, would be to have workshops, perhaps simultaneous work- 
shops, at the various levels of evaluation that Dr. Miller has mentioned. 

MISS CONRATH: I think it is appropriate that Regional Medical 
Programs may be the anchorman here. I would like to offer the sug- 
gestion and recommendation that you consider Regional Medical Programs 
as a mechanism for going the next step in the planning and carrying out 
of workshops along the lines of the suggestions; that have been made here 
for increasing experience and skills and evaluation as well as going 
farther in terms of learning how we can better coordinate evaluation 
procedures. 

I would like to make this recommendation, based on the obser- 
vation which has been made many different ways around the table, that 
probably the single biggest obstacle to the achievement of the potential 
of graduate and continuing education is that we have been hamstrung 
by the concepts and methods of traditional education of children, and 
that our language reflects this, our use of terms, how many times the 
word "courses, " "examinations, etc. , have been used. , , 

Until we learn how to become student- oriented and problem- 
oriented and learn to talk in terms of experiences and learn to really 
get invplyed much more in not who does the diagnosis of the educational 
.. problem or who. prescribes the treatment, but rather who participates 
in itp ; in recognizing that this is now a collaborative effort, .then, I 













think, we will begin to move much faster toward helping adults achieve 
the full potential of what the educational process has to offer. 

CHAIRMAN HALL: I think it is not fair to ask you to define 
your commonalities without making your own commitments. It seems 
to me it was first expressed by Elling, that there was foment today, 
the kind of foment that should create the change that we need to address 
ourselves to. 

The change should be forerun by the definition of objectives 
that are pragmatic. These objectives must relate to behavior changes, 
of understandable definitions, be real and have the capacity for com- 
mitment. They must be achievable and measurable. 

We discussed some of the ways and means and involvement of 
people in the measuring processes. We discussed the fact that these 
measuring processes and tools have to.be tailored to the individual 
needs of each situation. We have mentioned the fact that we have to 
define what universe we are going to work in before we can set up and 
choose the appropriate tools and obtain valid objectives. 

We heard how some people have effectively accepted this chal- 
lenge of defining their objectives and of using available tools, tailoring 
them to their specific needs, and I think in some respect tailoring them 
to the needs of the population. I felt a very strong trend in this con- 
ference through that as we did define our objectives, they were too 
often our objectives and not necessarily the people's objectives, that 
we were testing our needs rather than necessarily the people's needs 
or the learner's needs.' But I think that some people have addressed 
themselves to defining what the needs are and what the appropriate 
testing agents are and using these to relate back to the process. 

I think we also heard how just the development of these evalua- 
tion tools will permit and encourage a change in the attitude and behavior. 
This was particularly Dr. King's comment, I think, that he related how 
they had changed the behavior of the neurosurgeons as they went about 
addressing themselves to this problem of evaluation. 

: We heard about the need to gain confidence in the body to be 

evaluated, and steps and ways of doing this, and I think this is very 
valid, and one of the things that we have to address ourselves to. 

v We heard about the bodies that do assume responsibility for 
accreditation and evaluation programs and of people. We did not hear 
ve ry much about the s e bodie s as suming evaluation of proces s or making 
their evaluative materials available back to adjustment of the process. 
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We heard about what happens when a group is put under pres- 
sure. They either fold or react as a dynamic new program. They 
grasp the tools and define their position more effectively. 

We talked about the need for more relation to society and 
effective ways and the potentiality of the evaluation process and the 
endorsement process that goes on after an evaluation, of discrediting 
other means or mechanisms, and recognize that this was fraught with 
danger. 

We discussed the fact that evaluation should be a continuum 
like education. It is very important in our evaluation process to start 
early because sometimes we may have entered into our process a pro- 
duct that cannot receive and react appropriately to the process. 

We talked about the need for more resources in evaluation, both 
in people and in tools and what is being done by the federal agencies. 

We also talked about the responsibility of the agent of the people 
being the federal government to be sure that all systems and sub- 
systems of social institutions should provide proof of their legitimacy 
and their effectiveness in order to survive. 

We come up with recommendations, as I saw them: 

(1) To relate this more to the demand or needs of the people 
and the learners, for improved tools, for more dissemination of the 
kind of information that we had here today to our colleagues. We 
came up with the demand for action, for people to take back what we 
have gained here today to our substrait and see if we can truly be the 
enzyme that activates this substrait to be more effective. 

I think this is going to take commitment and involvement, and 
I would like to end with a little story that Dave tells so often about a 
Kamakazi pilot who flew twenty missions because he had commitment, 
but just couldn't get involved. ' 

I think you have got to get involved. 
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